Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11851

getGlobalJNIEnv() may deadlock if exception is thrown

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 3.0.0-alpha4
    • 3.0.0-alpha4
    • libhdfs
    • None

    Description

      HDFS-11529 introduced a deadlock into getGlobalJNIEnv() if an exception is thrown. getGlobalJNIEnv() holds jvmMutex, but printExceptionAndFree() will eventually try to acquire that lock in setTLSExceptionStrings().

      The exception might get caught from loadFileSystems:

      jthr = invokeMethod(env, NULL, STATIC, NULL,
                               "org/apache/hadoop/fs/FileSystem",
                               "loadFileSystems", "()V");
              if (jthr) {
                  printExceptionAndFree(env, jthr, PRINT_EXC_ALL, "loadFileSystems");
              }
          }
      

      and here's the relevant parts of the stack trace from where I call this API in Impala, which uses libhdfs:

      #0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
      #1  0x00007ffff4a8d657 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2  0x00007ffff4a8d480 in __GI___pthread_mutex_lock (mutex=0x47ce960 <jvmMutex>) at ../nptl/pthread_mutex_lock.c:79
      #3  0x0000000002f06056 in mutexLock (m=<optimized out>) at /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/os/posix/mutexes.c:28
      #4  0x0000000002efe817 in setTLSExceptionStrings (rootCause=0x0, stackTrace=0x0) at /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:581
      #5  0x0000000002f065d7 in printExceptionAndFreeV (env=0x513c1e8, exc=0x508a8c0, noPrintFlags=<optimized out>, fmt=0x34349cf "loadFileSystems", ap=0x7fffffffb660)
          at /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.c:183
      #6  0x0000000002f0683d in printExceptionAndFree (env=<optimized out>, exc=<optimized out>, noPrintFlags=<optimized out>, fmt=<optimized out>)
          at /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/exception.c:213
      #7  0x0000000002eff60f in getGlobalJNIEnv () at /data/2/jenkins/workspace/impala-hadoop-dependency/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:463
      

      Attachments

        1. HDFS-11851.005.patch
          3 kB
          Sailesh Mukil
        2. HDFS-11851.004.patch
          2 kB
          Sailesh Mukil
        3. HDFS-11851.003.patch
          3 kB
          Sailesh Mukil
        4. HDFS-11851.002.patch
          2 kB
          Sailesh Mukil
        5. HDFS-11851.001.patch
          0.7 kB
          Sailesh Mukil
        6. HDFS-11851.000.patch
          1 kB
          Sailesh Mukil

        Issue Links

          Activity

            People

              sailesh Sailesh Mukil
              henryr Henry Robinson
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: