[HDFS-14488] libhdfs SIGSEGV during shutdown of Java

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Cannot Reproduce
• Affects Version/s: 2.7.3, 3.2.0
• Component/s: native

Environment

Centos 7
gcc (GCC) 8.3.0

• Assignee: Unassigned
• Reporter: Kotomi
• Created: 13/May/19 03:53
• Updated: 09/Oct/19 07:48
• Resolved: 09/Oct/19 07:48

Description

For short, it‘s quite similar to https://issues.apache.org/jira/browse/HDFS-13585，but backtrace is different.

We are using libhdfs for hdfs support in our native library, as a plugin, dlopen when hdfs path detected and dlclose when application closes.

Crash happens randomly, during shutdown of Java. GDB shows in function hdfsThreadDestructor, JNIEnv v is not NULL but looks like a wild pointer. Therefore, it crashes when trying to get JavaVm from its own JNIEnv.

43: ret = (*env)->GetJavaVM(env, &vm);

Any workarounds? Thx.

Backtrace from core:

#1920 0x00007fac0386da47 in abort () from /usr/lib64/libc.so.6
#1921 0x00007fac0315b769 in os::abort(bool) () from /usr/java/jdk1.8.0_201-amd64/jre/lib/amd64/server/libjvm.so
#1922 0x00007fac03320803 in VMError::report_and_die() () from /usr/java/jdk1.8.0_201-amd64/jre/lib/amd64/server/libjvm.so
#1923 0x00007fac031659f5 in JVM_handle_linux_signal () from /usr/java/jdk1.8.0_201-amd64/jre/lib/amd64/server/libjvm.so
#1924 0x00007fac031588b8 in signalHandler(int, siginfo*, void*) () from /usr/java/jdk1.8.0_201-amd64/jre/lib/amd64/server/libjvm.so
#1925 <signal handler called>
#1926 0x00007fab028f7ef0 in hdfsThreadDestructor (v=0x7fab840919f8) at /home/ambari-qa/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/os/posix/thread_local_storage.c:43
#1927 0x00007fac04026bd2 in __nptl_deallocate_tsd () from /usr/lib64/libpthread.so.0
#1928 0x00007fac04026de3 in start_thread () from /usr/lib64/libpthread.so.0
#1929 0x00007fac03933ead in clone () from /usr/lib64/libc.so.6