[HDFS-13585] libhdfs SIGSEGV during shutdown of Java application.

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 2.7.5
• Component/s: native

Environment

Centos 7
gcc (GCC) 4.9.2 20150212 (Red Hat 4.9.2-6)

• Assignee: Unassigned
• Reporter: Nalini Ganapati
• Created: 17/May/18 21:04
• Updated: 06/Jul/23 01:11

Description

We are using libhdfs for hdfs support from our native library. This has been working mostly fine with Java/Spark applications, but some of them throw a SIGSEGV in hdfsThreadDestructor(). We tried to dynamically load and unload libhdfs.so using dlopen/dlclose but to no avail and we still see the seg fault. Is this a known issue? Looks like thread local storage is involved, are there workarounds? 

 

Here is a call stack from gdb java <core file>
(gdb) bt
#0 0x00007f3333ad21f7 in raise () from /usr/lib64/libc.so.6
#1 0x00007f3333ad38e8 in abort () from /usr/lib64/libc.so.6
#2 0x00007f3333380259 in os::abort(bool) () from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/jre/lib/amd64/server/libjvm.so
#3 0x00007f3333585986 in VMError::report_and_die() () from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/jre/lib/amd64/server/libjvm.so
#4 0x00007f3333389ec7 in JVM_handle_linux_signal () from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/jre/lib/amd64/server/libjvm.so
#5 0x00007f333337d678 in signalHandler(int, siginfo_t*, void*) () from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/jre/lib/amd64/server/libjvm.so
#6 <signal handler called>
#7 0x00007f3333341e66 in Monitor::ILock(Thread*) () from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/jre/lib/amd64/server/libjvm.so
#8 0x00007f33333428f6 in Monitor::lock_without_safepoint_check() () from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/jre/lib/amd64/server/libjvm.so
#9 0x00007f333358bc21 in VM_Exit::wait_if_vm_exited() () from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/jre/lib/amd64/server/libjvm.so
#10 0x00007f333314fee5 in jni_DetachCurrentThread () from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/jre/lib/amd64/server/libjvm.so
#11 0x00007f32f2645f15 in hdfsThreadDestructor (v=0x7f332c018bc8)
at /home/kshvachk/Work/Hadoop/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/os/posix/thread_local_storage.c:49
#12 0x00007f3334490c22 in __nptl_deallocate_tsd () from /usr/lib64/libpthread.so.0
#13 0x00007f3334490e33 in start_thread () from /usr/lib64/libpthread.so.0
#14 0x00007f3333b9534d in clone () from /usr/lib64/libc.so.6