Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Information Provided
-
Impala 2.7.0, Impala 2.7.1
-
None
-
None
-
# cat /etc/centos-release
CentOS release 6.9 (Final)
# cat /etc/grub.conf
default=0
timeout=1
title CentOS (2.6.32-696.3.2.el6.x86_64)
root (hd0)
kernel /boot/vmlinuz-2.6.32-696.3.2.el6.x86_64 root=LABEL=centos_root ro crashkernel=auto LANG=en_US.UTF-8 KEYTABLE=us
initrd /boot/initramfs-2.6.32-696.3.2.el6.x86_64.img# cat /etc/centos-release CentOS release 6.9 (Final) # cat /etc/grub.conf default=0 timeout=1 title CentOS (2.6.32-696.3.2.el6.x86_64) root (hd0) kernel /boot/vmlinuz-2.6.32-696.3.2.el6.x86_64 root=LABEL=centos_root ro crashkernel=auto LANG=en_US.UTF-8 KEYTABLE=us initrd /boot/initramfs-2.6.32-696.3.2.el6.x86_64.img
-
ghx-label-1
Description
I had a working configuration (Impala Shell v2.7.0-cdh5.10.1) and updated the Linux kernel from "2.6.32-642.4.2.el6" to "2.6.32-696.3.2.el6" as below:
Installing: kernel x86_64 2.6.32-696.3.2.el6 updates 32 M kernel-devel x86_64 2.6.32-696.3.2.el6 updates 11 M Updating: java-1.7.0-openjdk x86_64 1:1.7.0.141-2.6.10.1.el6_9 updates 26 M java-1.7.0-openjdk-devel x86_64 1:1.7.0.141-2.6.10.1.el6_9 updates 9.5 M kernel-firmware noarch 2.6.32-696.3.2.el6 updates 29 M kernel-headers x86_64 2.6.32-696.3.2.el6 updates 4.5 M
After this impala services crash at startup, this does not happen on CDH5.9, but does happen with CDH 5.10.0 and 5.10.1.
# service impala-catalog status Impala Catalog Server is dead and pid file exists [FAILED] # service impala-server status Impala Server is dead and pid file exists [FAILED] # service impala-state-store status Impala State Store Server is running [ OK ]
In impalad.ip-10-45-1-67.impala.log.INFO.20170626-120509.3044 I see:
# # A fatal error has been detected by the Java Runtime Environment: # # SIGBUS (0x7) at pc=0x00007f028e8e046f, pid=3044, tid=139649127274624 # # JRE version: (7.0_71-b14) (build ) # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.71-b01 mixed mode linux-amd64 compressed oops) # Problematic frame: # j java.lang.Object.<clinit>()V+0 # # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again # # An error report file with more information is saved as: # /var/lib/impala/hs_err_pid3044.log
I've uploaded hs_err_pid3044.log for reference.
If I switch to the previous kernel and restart by changing grub.conf "default" to "1" as below then impala services start successfully:
# cat /etc/grub.conf
default=0
timeout=1
title CentOS (2.6.32-696.3.2.el6.x86_64)
root (hd0)
kernel /boot/vmlinuz-2.6.32-696.3.2.el6.x86_64 root=LABEL=centos_root ro crashkernel=auto LANG=en_US.UTF-8 KEYTABLE=us
initrd /boot/initramfs-2.6.32-696.3.2.el6.x86_64.img
title CentOS (2.6.32-642.4.2.el6.x86_64)
root (hd0)
kernel /boot/vmlinuz-2.6.32-642.4.2.el6.x86_64 root=LABEL=centos_root ro crashkernel=auto LANG=en_US.UTF-8 KEYTABLE=us
initrd /boot/initramfs-2.6.32-642.4.2.el6.x86_64.img
# shutdown -r now
Likewise if I repeat the above with "default=0" then the issue returns.