Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14214

DomainSocketWatcher::add()/delete() should not self interrupt while looping await()

    Details

      Description

      Our hive team found a TPCDS job whose queries running on LLAP seem to be getting stuck. Dozens of threads were waiting for the DfsClientShmManager::lock, as following jstack:

      Thread 251 (IO-Elevator-Thread-5):
        State: WAITING
        Blocked count: 3871
        Wtaited count: 4565
        Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@16ead198
        Stack:
          sun.misc.Unsafe.park(Native Method)
          java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
          java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1976)
          org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager.allocSlot(DfsClientShmManager.java:255)
          org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager.allocSlot(DfsClientShmManager.java:434)
          org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.allocShmSlot(ShortCircuitCache.java:1017)
          org.apache.hadoop.hdfs.BlockReaderFactory.createShortCircuitReplicaInfo(BlockReaderFactory.java:476)
          org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.create(ShortCircuitCache.java:784)
          org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.fetchOrCreate(ShortCircuitCache.java:718)
          org.apache.hadoop.hdfs.BlockReaderFactory.getBlockReaderLocal(BlockReaderFactory.java:422)
          org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:333)
          org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1181)
          org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:1118)
          org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1478)
          org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1441)
          org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:121)
          org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:111)
          org.apache.orc.impl.RecordReaderUtils$DefaultDataReader.readStripeFooter(RecordReaderUtils.java:166)
          org.apache.hadoop.hive.llap.io.metadata.OrcStripeMetadata.<init>(OrcStripeMetadata.java:64)
          org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.readStripesMetadata(OrcEncodedDataReader.java:622)
      

      The thread that is expected to signal those threads is calling DomainSocketWatcher::add() method, but it gets stuck there dealing with InterruptedException infinitely. The jstack is like:

      Thread 44417 (TezTR-257387_2840_12_10_52_0):
        State: RUNNABLE
        Blocked count: 3
        Wtaited count: 5
        Stack:
          java.lang.Throwable.fillInStackTrace(Native Method)
          java.lang.Throwable.fillInStackTrace(Throwable.java:783)
          java.lang.Throwable.<init>(Throwable.java:250)
          java.lang.Exception.<init>(Exception.java:54)
          java.lang.InterruptedException.<init>(InterruptedException.java:57)
          java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2034)
          org.apache.hadoop.net.unix.DomainSocketWatcher.add(DomainSocketWatcher.java:325)
          org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager.allocSlot(DfsClientShmManager.java:266)
          org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager.allocSlot(DfsClientShmManager.java:434)
          org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.allocShmSlot(ShortCircuitCache.java:1017)
          org.apache.hadoop.hdfs.BlockReaderFactory.createShortCircuitReplicaInfo(BlockReaderFactory.java:476)
          org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.create(ShortCircuitCache.java:784)
          org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.fetchOrCreate(ShortCircuitCache.java:718)
          org.apache.hadoop.hdfs.BlockReaderFactory.getBlockReaderLocal(BlockReaderFactory.java:422)
          org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:333)
          org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1181)
          org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:1118)
          org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1478)
          org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1441)
          org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:121)
      

      The whole job makes no progress because of this.

      The thread in DomainSocketWatcher::add() is expected to eventually break the while loop where it waits for the newly added entry being deleted by another thread. However, if this thread is ever interrupted, chances are that it will hold the lock forever so if(!toAdd.contains(entry)) always be false.

      DomainSocketWatcher::add()
        public void add(DomainSocket sock, Handler handler) {
          lock.lock();
          try {
            ......
            toAdd.add(entry);
            kick();
            while (true) {
              try {
                processedCond.await();
              } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
              }
              if (!toAdd.contains(entry)) {
                break;
              }
            }
          } finally {
            lock.unlock();
          }
        }
      

      The reason here is that, this method catches the InterruptedException and self interrupts during await(). The await() method internally calls AbstractQueuedSynchronizer::await(), which will throw a new InterruptedException if it's interrupted.

      AbstractQueuedSynchronizer::await()
              public final void await() throws InterruptedException {
                  if (Thread.interrupted())
                      throw new InterruptedException();
                  Node node = addConditionWaiter();
                  ...
      

      Our code in DomainSocketWatcher::add() catches this exception (again) and self interrupt (again). Please note in this process, the associated lock is never released so that the other thread which is supposed to make if(!toAdd.contains(entry)) be true is still pending on the lock.

      DomainSocketWatcher::delete() has similar code logic and should suffer from similar problems.

      Thanks Jason Dere for testing and reporting this.

        Issue Links

          Activity

          Hide
          zhz Zhe Zhang added a comment -

          Thanks Mingliang Liu for the fix. Indeed a major bug. I just backported to branch-2.7.

          Show
          zhz Zhe Zhang added a comment - Thanks Mingliang Liu for the fix. Indeed a major bug. I just backported to branch-2.7.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - 2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11450 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11450/)
          HADOOP-14214. DomainSocketWatcher::add()/delete() should not self (liuml07: rev d35e79abc2fee7153a6168e6088f100de59d8c81)

          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/unix/DomainSocketWatcher.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11450 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11450/ ) HADOOP-14214 . DomainSocketWatcher::add()/delete() should not self (liuml07: rev d35e79abc2fee7153a6168e6088f100de59d8c81) (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/unix/DomainSocketWatcher.java
          Hide
          liuml07 Mingliang Liu added a comment -

          Committed to trunk through branch-2.8 branches. Thanks Sergey Shelukhin for discussion. Thanks Jitendra Nath Pandey and Tsz Wo Nicholas Sze for review.

          Show
          liuml07 Mingliang Liu added a comment - Committed to trunk through branch-2.8 branches. Thanks Sergey Shelukhin for discussion. Thanks Jitendra Nath Pandey and Tsz Wo Nicholas Sze for review.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          The try-await-catch-InterruptedException-interrupt is clearly a bug. Using awaitUninterruptibly sounds good.

          +1 on the patch.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - The try-await-catch-InterruptedException-interrupt is clearly a bug. Using awaitUninterruptibly sounds good. +1 on the patch.
          Hide
          jnp Jitendra Nath Pandey added a comment -

          The wait is to ensure the domain-socket fd is added to the DomainSocketWatcher thread's collection. It is a very short wait, but important so that fd's are managed appropriately. The interrupt is not lost, awaitUninterruptibly ensures interrupt is set when it returns.

          Show
          jnp Jitendra Nath Pandey added a comment - The wait is to ensure the domain-socket fd is added to the DomainSocketWatcher thread's collection. It is a very short wait, but important so that fd's are managed appropriately. The interrupt is not lost, awaitUninterruptibly ensures interrupt is set when it returns.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 25s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 12m 39s trunk passed
          +1 compile 20m 17s trunk passed
          +1 checkstyle 0m 35s trunk passed
          +1 mvnsite 1m 0s trunk passed
          +1 mvneclipse 0m 19s trunk passed
          +1 findbugs 1m 25s trunk passed
          +1 javadoc 0m 50s trunk passed
          +1 mvninstall 0m 37s the patch passed
          +1 compile 15m 58s the patch passed
          +1 javac 15m 58s the patch passed
          +1 checkstyle 0m 36s the patch passed
          +1 mvnsite 1m 0s the patch passed
          +1 mvneclipse 0m 19s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 38s the patch passed
          +1 javadoc 0m 49s the patch passed
          -1 unit 7m 44s hadoop-common in the patch failed.
          +1 asflicense 0m 33s The patch does not generate ASF License warnings.
          68m 36s



          Reason Tests
          Failed junit tests hadoop.security.TestRaceWhenRelogin



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-14214
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12860055/HADOOP-14214.000.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux a20fbdc56866 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f462e1f
          Default Java 1.8.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11889/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11889/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11889/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 25s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 12m 39s trunk passed +1 compile 20m 17s trunk passed +1 checkstyle 0m 35s trunk passed +1 mvnsite 1m 0s trunk passed +1 mvneclipse 0m 19s trunk passed +1 findbugs 1m 25s trunk passed +1 javadoc 0m 50s trunk passed +1 mvninstall 0m 37s the patch passed +1 compile 15m 58s the patch passed +1 javac 15m 58s the patch passed +1 checkstyle 0m 36s the patch passed +1 mvnsite 1m 0s the patch passed +1 mvneclipse 0m 19s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 38s the patch passed +1 javadoc 0m 49s the patch passed -1 unit 7m 44s hadoop-common in the patch failed. +1 asflicense 0m 33s The patch does not generate ASF License warnings. 68m 36s Reason Tests Failed junit tests hadoop.security.TestRaceWhenRelogin Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-14214 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12860055/HADOOP-14214.000.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a20fbdc56866 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f462e1f Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11889/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11889/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11889/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          sershe Sergey Shelukhin added a comment - - edited

          Hmm.. the reason we are interrupting the thread in question is because we want it to be interrupted (because the work it's performing is no longer relevant). Wouldn't this just cause it to be stuck forever anyway, or at best to continue a useless operation? cc Siddharth Seth

          Show
          sershe Sergey Shelukhin added a comment - - edited Hmm.. the reason we are interrupting the thread in question is because we want it to be interrupted (because the work it's performing is no longer relevant). Wouldn't this just cause it to be stuck forever anyway, or at best to continue a useless operation? cc Siddharth Seth
          Hide
          liuml07 Mingliang Liu added a comment -
          Show
          liuml07 Mingliang Liu added a comment - Ping Colin P. McCabe , Jitendra Nath Pandey , Arpit Agarwal for discussion.
          Hide
          liuml07 Mingliang Liu added a comment -

          One possible fix is to call awaitUninterruptibly() instead of await().

          Show
          liuml07 Mingliang Liu added a comment - One possible fix is to call awaitUninterruptibly() instead of await() .

            People

            • Assignee:
              liuml07 Mingliang Liu
              Reporter:
              liuml07 Mingliang Liu
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development