Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-723

Deadlock in DFSClient#DFSOutputStream

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.20.2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      WhiIe was running some append-related tests, I hit this deadlock:

      Found one Java-level deadlock:
      =============================
      "Thread-3":
        waiting to lock monitor 0x000000012ee044f0 (object 0x0000000107a0ded0, a org.apache.hadoop.hdfs.DFSClient$DFSOutputStream),
        which is held by "main"
      "main":
        waiting to lock monitor 0x000000012eeb71a8 (object 0x00000001082b0748, a org.apache.hadoop.hdfs.DFSClient$LeaseChecker),
        which is held by "Thread-3"
      
      Java stack information for the threads listed above:
      ===================================================
      "Thread-3":
      	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3582)
      	- waiting to lock <0x0000000107a0ded0> (a org.apache.hadoop.hdfs.DFSClient$DFSOutputStream)
      	at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.close(DFSClient.java:1175)
      	- locked <0x00000001082b0748> (a org.apache.hadoop.hdfs.DFSClient$LeaseChecker)
      	at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:306)
      	- locked <0x000000010824d640> (a org.apache.hadoop.hdfs.DFSClient)
              - waiting to lock <0x0000000107a0ded0> (a org.apache.hadoop.hdfs.DFSClient$DFSOutputStream)
              at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.close(DFSClient.java:1175)
              - locked <0x00000001082b0748> (a org.apache.hadoop.hdfs.DFSClient$LeaseChecker)
              at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:306)
              - locked <0x000000010824d640> (a org.apache.hadoop.hdfs.DFSClient)
              at org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:325)
              at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:1835)
              - locked <0x0000000107a77ec8> (a org.apache.hadoop.fs.FileSystem$Cache)
              at org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:1851)
              - locked <0x00000001079daa00> (a org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer)
      "main":
              at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.remove(DFSClient.java:1151)
              - waiting to lock <0x00000001082b0748> (a org.apache.hadoop.hdfs.DFSClient$LeaseChecker)
              at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3609)
              - locked <0x0000000107a0ded0> (a org.apache.hadoop.hdfs.DFSClient$DFSOutputStream)
              at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
              at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
              at org.apache.hadoop.hdfs.TestFileAppend4.testAppend(TestFileAppend4.java:99)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at junit.framework.TestCase.runTest(TestCase.java:168)
              at junit.framework.TestCase.runBare(TestCase.java:134)
              at junit.framework.TestResult$1.protect(TestResult.java:110)
              at junit.framework.TestResult.runProtected(TestResult.java:128)
              at junit.framework.TestResult.run(TestResult.java:113)
              at junit.framework.TestCase.run(TestCase.java:124)
              at junit.framework.TestSuite.runTest(TestSuite.java:232)
              at junit.framework.TestSuite.run(TestSuite.java:227)
              at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
              at junit.extensions.TestSetup$1.protect(TestSetup.java:23)
              at junit.framework.TestResult.runProtected(TestResult.java:128)
              at junit.extensions.TestSetup.run(TestSetup.java:27)
              at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420)
              at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911)
              at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768)
      
      1. deadlock_0.21.patch
        0.8 kB
        Hairong Kuang
      2. deadlock.patch
        0.8 kB
        Hairong Kuang

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            Hairong Kuang
            Reporter:
            Hairong Kuang
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development