[HDFS-12494] libhdfs SIGSEGV in setTLSExceptionStrings

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha4
• Fix Version/s: 3.0.0, 3.1.0
• Component/s: libhdfs
• Target Version/s: 3.0.0
• Assignee: John Zhuge
• Reporter: John Zhuge
• Created: 19/Sep/17 17:38
• Updated: 07/Oct/17 07:22
• Resolved: 07/Oct/17 07:10

Description

libhdfs application crashes when CLASSPATH is set but not set properly. It uses wildcard in this case.

$ export CLASSPATH=$(hadoop classpath)
$ pwd
/Users/jzhuge/hadoop2/hadoop-hdfs-project/hadoop-hdfs/target/native

$ ./test_libhdfs_ops
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00000001052968f7, pid=14147, tid=775
#
# JRE version: Java(TM) SE Runtime Environment (7.0_79-b15) (build 1.7.0_79-b15)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.79-b02 mixed mode bsd-amd64 compressed oops)
# Problematic frame:
# C  [libhdfs.0.0.0.dylib+0x38f7]  setTLSExceptionStrings+0x47
#
# Core dump written. Default location: /cores/core or core.14147
#
# An error report file with more information is saved as:
# /Users/jzhuge/hadoop2/hadoop-hdfs-project/hadoop-hdfs/target/native/hs_err_pid14147.log
#
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
Abort trap: 6 (core dumped)

[jzhuge@jzhuge-MBP native]((be32925fff5...) *+)$ lldb -c /cores/core.14147
(lldb) target create --core "/cores/core.14147"
warning: (x86_64) /cores/core.14147 load command 549 LC_SEGMENT_64 has a fileoff + filesize (0x14627f000) that extends beyond the end of the file (0x14627e000), the segment will be truncated to match
warning: (x86_64) /cores/core.14147 load command 550 LC_SEGMENT_64 has a fileoff (0x14627f000) that extends beyond the end of the file (0x14627e000), ignoring this section
Core file '/cores/core.14147' (x86_64) was loaded.
(lldb) bt
* thread #1, stop reason = signal SIGSTOP
  * frame #0: 0x00007fffcf89ad42 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fffcf988457 libsystem_pthread.dylib`pthread_kill + 90
    frame #2: 0x00007fffcf800420 libsystem_c.dylib`abort + 129
    frame #3: 0x00000001056cd5fb libjvm.dylib`os::abort(bool) + 25
    frame #4: 0x00000001057d98fc libjvm.dylib`VMError::report_and_die() + 2308
    frame #5: 0x00000001056cefb5 libjvm.dylib`JVM_handle_bsd_signal + 1083
    frame #6: 0x00007fffcf97bb3a libsystem_platform.dylib`_sigtramp + 26
    frame #7: 0x00000001052968f8 libhdfs.0.0.0.dylib`setTLSExceptionStrings(rootCause=0x0000000000000000, stackTrace=0x0000000000000000) at jni_helper.c:589 [opt]
    frame #8: 0x00000001052954f0 libhdfs.0.0.0.dylib`printExceptionAndFreeV(env=0x00007ffaff0019e8, exc=0x00007ffafec04140, noPrintFlags=<unavailable>, fmt="loadFileSystems", ap=<unavailable>) at exception.c:183 [opt]
    frame #9: 0x00000001052956bb libhdfs.0.0.0.dylib`printExceptionAndFree(env=<unavailable>, exc=<unavailable>, noPrintFlags=<unavailable>, fmt=<unavailable>) at exception.c:213 [opt]
    frame #10: 0x00000001052967f4 libhdfs.0.0.0.dylib`getJNIEnv [inlined] getGlobalJNIEnv at jni_helper.c:463 [opt]
    frame #11: 0x000000010529664f libhdfs.0.0.0.dylib`getJNIEnv at jni_helper.c:528 [opt]
    frame #12: 0x00000001052975eb libhdfs.0.0.0.dylib`hdfsBuilderConnect(bld=0x00007ffafed00000) at hdfs.c:693 [opt]
    frame #13: 0x000000010528be30 test_libhdfs_ops`main(argc=<unavailable>, argv=<unavailable>) at test_libhdfs_ops.c:91 [opt]
    frame #14: 0x00007fffcf76c235 libdyld.dylib`start + 1
(lldb) f 10
libhdfs.0.0.0.dylib was compiled with optimization - stepping may behave oddly; variables may not be available.
frame #10: 0x00000001052967f4 libhdfs.0.0.0.dylib`getJNIEnv [inlined] getGlobalJNIEnv at jni_helper.c:463 [opt]
   460 	                         "org/apache/hadoop/fs/FileSystem",
   461 	                         "loadFileSystems", "()V");
   462 	        if (jthr) {
-> 463 	            printExceptionAndFree(env, jthr, PRINT_EXC_ALL, "loadFileSystems");
   464 	        }
   465 	    }
   466 	    else {
(lldb) f 7
frame #7: 0x00000001052968f8 libhdfs.0.0.0.dylib`setTLSExceptionStrings(rootCause=0x0000000000000000, stackTrace=0x0000000000000000) at jni_helper.c:589 [opt]
   586 	        mutexUnlock(&jvmMutex);
   587 	    }
   588
-> 589 	    free(state->lastExceptionRootCause);
   590 	    free(state->lastExceptionStackTrace);
   591 	    state->lastExceptionRootCause = (char*)rootCause;
   592 	    state->lastExceptionStackTrace = (char*)stackTrace;
(lldb) p state
(ThreadLocalState *) $0 = 0x0000000000000000

The correct way to set CLASSPATH is because libhdfs does not support wildcard in CLASSPATH:

$ export CLASSPATH=$(hadoop classpath --glob)

Filed HDFS-12491 Support wildcard in CLASSPATH for libhdfs.

Issue Links

• is broken by: HDFS-11529 Add libHDFS API to return last exception
• relates to: HDFS-11851 getGlobalJNIEnv() may deadlock if exception is thrown