[IMPALA-7150] Crash in Reflection::invoke_method()

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.13.0, Impala 3.1.0
• Fix Version/s: Impala 2.13.0, Impala 3.1.0
• Component/s: Backend, Frontend
• Labels: broken-build crash flaky
• Epic Color: ghx-label-2
• Assignee: Vuk Ercegovac
• Reporter: Lars Volker
• Created: 08/Jun/18 00:27
• Updated: 12/Jul/18 19:02
• Resolved: 21/Jun/18 01:28

Description

We've seen several crashes in the past involving Reflection::invoke_method(). The stack trace looks like this:

CORE: ./core.1527786725.23658.impalad
BINARY: ./be/build/latest/service/impalad
Core was generated by `/data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/be/buil'.
Program terminated with signal 6, Aborted.
#0  0x00007fca4ff4a5c9 in raise () from /lib64/libc.so.6
To enable execution of this file add
	add-auto-load-safe-path /data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/Impala-Toolchain/gcc-4.9.2/lib64/libstdc++.so.6.0.20-gdb.py
line to your configuration file "/var/lib/jenkins/.gdbinit".
To completely disable this security protection add
	set auto-load safe-path /
line to your configuration file "/var/lib/jenkins/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
	info "(gdb)Auto-loading safe path"
#0  0x00007fca4ff4a5c9 in raise () from /lib64/libc.so.6
#1  0x00007fca4ff4bcd8 in abort () from /lib64/libc.so.6
#2  0x00007fca52db61a5 in os::abort(bool) () from /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#3  0x00007fca52f46843 in VMError::report_and_die() () from /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#4  0x00007fca52dbb562 in JVM_handle_linux_signal () from /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#5  0x00007fca52db24f3 in signalHandler(int, siginfo*, void*) () from /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#6  <signal handler called>
#7  0x00007fca52e29621 in Reflection::invoke(instanceKlassHandle, methodHandle, Handle, bool, objArrayHandle, BasicType, objArrayHandle, bool, Thread*) () from /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#8  0x00007fca52e2cd27 in Reflection::invoke_method(oopDesc*, Handle, objArrayHandle, Thread*) () from /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#9  0x00007fca52bc339b in JVM_InvokeMethod () from /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#10 0x00007fca36f49d6d in ?? ()
#11 0x00000006600753f0 in ?? ()
#12 0x000000066006f000 in ?? ()
#13 0x0000000660079328 in ?? ()
#14 0x000000065f3e6550 in ?? ()
#15 0x00000007c0733028 in ?? ()
#16 0x00007fc98ff20bf0 in ?? ()
#17 0x000000039d85edd0 in ?? ()
#18 0x00007fca36f48cb6 in ?? ()
#19 0x00000006600753f0 in ?? ()
#20 0x00007fca36f48f0c in ?? ()
#21 0x0000001400000001 in ?? ()
#22 0x0000000000000000 in ?? ()

From the core file I was unable to figure out what had happened. Unfortunately we also did not preserver the hs_err_pid* files until recently (fixed here).

We should track this down when it happens again.

Issue Links

• causes: IMPALA-7069 Java UDF tests can trigger a crash in Java ClassLoader
• is broken by: IMPALA-6215 Race between lib_cache and java udf class loading