[IMPALA-5578] Impala services crash after Linux kernel update

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Information Provided
• Affects Version/s: Impala 2.7.0, Impala 2.7.1
• Component/s: Infrastructure

Environment

# cat /etc/centos-release
CentOS release 6.9 (Final)

# cat /etc/grub.conf
default=0
timeout=1

title CentOS (2.6.32-696.3.2.el6.x86_64)
        root (hd0)
        kernel /boot/vmlinuz-2.6.32-696.3.2.el6.x86_64 root=LABEL=centos_root ro crashkernel=auto LANG=en_US.UTF-8 KEYTABLE=us
        initrd /boot/initramfs-2.6.32-696.3.2.el6.x86_64.img

• Epic Color: ghx-label-1
• Assignee: Unassigned
• Reporter: Neil Johnson
• Created: 26/Jun/17 12:25
• Updated: 24/Aug/17 21:38
• Resolved: 24/Aug/17 21:38

Description

I had a working configuration (Impala Shell v2.7.0-cdh5.10.1) and updated the Linux kernel from "2.6.32-642.4.2.el6" to "2.6.32-696.3.2.el6" as below:

Installing:
 kernel                                   x86_64                 2.6.32-696.3.2.el6                            updates                  32 M
 kernel-devel                             x86_64                 2.6.32-696.3.2.el6                            updates                  11 M
Updating:
 java-1.7.0-openjdk                       x86_64                 1:1.7.0.141-2.6.10.1.el6_9                    updates                  26 M
 java-1.7.0-openjdk-devel                 x86_64                 1:1.7.0.141-2.6.10.1.el6_9                    updates                 9.5 M
 kernel-firmware                          noarch                 2.6.32-696.3.2.el6                            updates                  29 M
 kernel-headers                           x86_64                 2.6.32-696.3.2.el6                            updates                 4.5 M

After this impala services crash at startup, this does not happen on CDH5.9, but does happen with CDH 5.10.0 and 5.10.1.

# service impala-catalog status
Impala Catalog Server is dead and pid file exists          [FAILED]
# service impala-server status
Impala Server is dead and pid file exists                  [FAILED]
# service impala-state-store status
Impala State Store Server is running                       [  OK  ]

In impalad.ip-10-45-1-67.impala.log.INFO.20170626-120509.3044 I see:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0x7) at pc=0x00007f028e8e046f, pid=3044, tid=139649127274624
#
# JRE version:  (7.0_71-b14) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.71-b01 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# j  java.lang.Object.<clinit>()V+0
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /var/lib/impala/hs_err_pid3044.log

I've uploaded hs_err_pid3044.log for reference.

If I switch to the previous kernel and restart by changing grub.conf "default" to "1" as below then impala services start successfully:

# cat /etc/grub.conf
default=0
timeout=1

title CentOS (2.6.32-696.3.2.el6.x86_64)
        root (hd0)
        kernel /boot/vmlinuz-2.6.32-696.3.2.el6.x86_64 root=LABEL=centos_root ro crashkernel=auto LANG=en_US.UTF-8 KEYTABLE=us
        initrd /boot/initramfs-2.6.32-696.3.2.el6.x86_64.img
title CentOS (2.6.32-642.4.2.el6.x86_64)
        root (hd0)
        kernel /boot/vmlinuz-2.6.32-642.4.2.el6.x86_64 root=LABEL=centos_root ro crashkernel=auto LANG=en_US.UTF-8 KEYTABLE=us
        initrd /boot/initramfs-2.6.32-642.4.2.el6.x86_64.img
# shutdown -r now

Likewise if I repeat the above with "default=0" then the issue returns.