Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9294

DFSClient deadlock when close file and failed to renew lease

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.2.0, 2.7.1
    • Fix Version/s: 2.8.0, 2.7.2, 2.6.4, 3.0.0-alpha1
    • Component/s: hdfs-client
    • Labels:
      None
    • Environment:

      Hadoop 2.2.0

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      We found a deadlock at our HBase(0.98) cluster(and the Hadoop Version is 2.2.0),and it should be HDFS BUG,at the time our network is not stable.
      below is the stack:

      *************************************************************************************************************************************
      Found one Java-level deadlock:
      =============================
      "MemStoreFlusher.1":
      waiting to lock monitor 0x00007ff27cfa5218 (object 0x00000002fae5ebe0, a org.apache.hadoop.hdfs.LeaseRenewer),
      which is held by "LeaseRenewer:hbaseadmin@hbase-ns-gdt-sh-marvel"
      "LeaseRenewer:hbaseadmin@hbase-ns-gdt-sh-marvel":
      waiting to lock monitor 0x00007ff2e67e16a8 (object 0x0000000486ce6620, a org.apache.hadoop.hdfs.DFSOutputStream),
      which is held by "MemStoreFlusher.0"
      "MemStoreFlusher.0":
      waiting to lock monitor 0x00007ff27cfa5218 (object 0x00000002fae5ebe0, a org.apache.hadoop.hdfs.LeaseRenewer),
      which is held by "LeaseRenewer:hbaseadmin@hbase-ns-gdt-sh-marvel"

      Java stack information for the threads listed above:
      ===================================================
      "MemStoreFlusher.1":
      at org.apache.hadoop.hdfs.LeaseRenewer.addClient(LeaseRenewer.java:216)

      • waiting to lock <0x00000002fae5ebe0> (a org.apache.hadoop.hdfs.LeaseRenewer)
        at org.apache.hadoop.hdfs.LeaseRenewer.getInstance(LeaseRenewer.java:81)
        at org.apache.hadoop.hdfs.DFSClient.getLeaseRenewer(DFSClient.java:648)
        at org.apache.hadoop.hdfs.DFSClient.endFileLease(DFSClient.java:659)
        at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:1882)
      • locked <0x000000055b606cb0> (a org.apache.hadoop.hdfs.DFSOutputStream)
        at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:71)
        at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:104)
        at org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter.finishClose(AbstractHFileWriter.java:250)
        at org.apache.hadoop.hbase.io.hfile.HFileWriterV2.close(HFileWriterV2.java:402)
        at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.close(StoreFile.java:974)
        at org.apache.hadoop.hbase.regionserver.StoreFlusher.finalizeWriter(StoreFlusher.java:78)
        at org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultStoreFlusher.java:75)
      • locked <0x000000059869eed8> (a java.lang.Object)
        at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:812)
        at org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.java:1974)
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1795)
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1678)
        at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1591)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:472)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:211)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$500(MemStoreFlusher.java:66)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:238)
        at java.lang.Thread.run(Thread.java:744)
        "LeaseRenewer:hbaseadmin@hbase-ns-gdt-sh-marvel":
        at org.apache.hadoop.hdfs.DFSOutputStream.abort(DFSOutputStream.java:1822)
      • waiting to lock <0x0000000486ce6620> (a org.apache.hadoop.hdfs.DFSOutputStream)
        at org.apache.hadoop.hdfs.DFSClient.closeAllFilesBeingWritten(DFSClient.java:780)
        at org.apache.hadoop.hdfs.DFSClient.abort(DFSClient.java:753)
        at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:453)
      • locked <0x00000002fae5ebe0> (a org.apache.hadoop.hdfs.LeaseRenewer)
        at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
        at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
        at java.lang.Thread.run(Thread.java:744)
        "MemStoreFlusher.0":
        at org.apache.hadoop.hdfs.LeaseRenewer.addClient(LeaseRenewer.java:216)
      • waiting to lock <0x00000002fae5ebe0> (a org.apache.hadoop.hdfs.LeaseRenewer)
        at org.apache.hadoop.hdfs.LeaseRenewer.getInstance(LeaseRenewer.java:81)
        at org.apache.hadoop.hdfs.DFSClient.getLeaseRenewer(DFSClient.java:648)
        at org.apache.hadoop.hdfs.DFSClient.endFileLease(DFSClient.java:659)
        at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:1882)
      • locked <0x0000000486ce6620> (a org.apache.hadoop.hdfs.DFSOutputStream)
        at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:71)
        at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:104)
        at org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter.finishClose(AbstractHFileWriter.java:250)
        at org.apache.hadoop.hbase.io.hfile.HFileWriterV2.close(HFileWriterV2.java:402)
        at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.close(StoreFile.java:974)
        at org.apache.hadoop.hbase.regionserver.StoreFlusher.finalizeWriter(StoreFlusher.java:78)
        at org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultStoreFlusher.java:75)
      • locked <0x00000004888f6848> (a java.lang.Object)
        at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:812)
        at org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.java:1974)
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1795)
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1678)
        at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1591)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:472)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:435)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$800(MemStoreFlusher.java:66)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:253)
        at java.lang.Thread.run(Thread.java:744)

      Found 1 deadlock.

      **********************************************************************

      the thread "MemStoreFlusher.0" is closing outputStream and remove it's lease ;
      other side the daemon thread "LeaseRenewer" failed to connect active nn for renewing lease,but got SocketTimeoutException cause of network is not good,so abort outputstream.
      then deadlock is made.

      and it seems not solved at Hadoop 2.7.1 .If confirmed , we can fixed the issue.

      1. HDFS-9294.patch
        3 kB
        Brahma Reddy Battula
      2. HDFS-9294-002.patch
        4 kB
        Brahma Reddy Battula
      3. HDFS-9294-branch-2.7.patch
        2 kB
        Brahma Reddy Battula
      4. HDFS-9294-branch-2.patch
        2 kB
        Brahma Reddy Battula
      5. HDFS-9294-002.patch
        4 kB
        Tsz Wo Nicholas Sze
      6. HDFS-9294-branch-2.6.patch
        4 kB
        Brahma Reddy Battula

        Issue Links

          Activity

          Hide
          djp Junping Du added a comment -

          BTW, I set back fix version to include 2.7.2 and 2.8.0. Given 2.6.4 will be released after 2.7.2, we are not assuming all fixes in 2.6.4 will be included in 2.7.2. FYI.

          Show
          djp Junping Du added a comment - BTW, I set back fix version to include 2.7.2 and 2.8.0. Given 2.6.4 will be released after 2.7.2, we are not assuming all fixes in 2.6.4 will be included in 2.7.2. FYI.
          Hide
          djp Junping Du added a comment -

          Thanks Brahma Reddy Battula and Tsz Wo Nicholas Sze for help on this!

          Show
          djp Junping Du added a comment - Thanks Brahma Reddy Battula and Tsz Wo Nicholas Sze for help on this!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9126 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9126/)
          In CHANGES.txt, move HDFS-9294 to Release 2.6.4. (szetszwo: rev 995f06f2684cd30efc5d12d924dda42879232099)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9126 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9126/ ) In CHANGES.txt, move HDFS-9294 to Release 2.6.4. (szetszwo: rev 995f06f2684cd30efc5d12d924dda42879232099) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          I have committed the branch-2.6 patch. Thanks, Brahma Reddy Battula!

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - I have committed the branch-2.6 patch. Thanks, Brahma Reddy Battula !
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          +1 the branch-2.6 patch looks good.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - +1 the branch-2.6 patch looks good.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Junping Du and Tsz Wo Nicholas Sze uploaded the branch-2.6 patch...

          Show
          brahmareddy Brahma Reddy Battula added a comment - Junping Du and Tsz Wo Nicholas Sze uploaded the branch-2.6 patch...
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          will upload the patch soon for branch-2.6..

          Show
          brahmareddy Brahma Reddy Battula added a comment - will upload the patch soon for branch-2.6..
          Hide
          djp Junping Du added a comment -

          Thanks for your confirmation, Nicholas. I tried to cherry-pick to branch-2.6, however, it have several conflicts. Brahma Reddy Battula and Tsz Wo Nicholas Sze, would you help to merge the commit to branch-2.6 or deliver a 2.6 patch here? Thanks!

          Show
          djp Junping Du added a comment - Thanks for your confirmation, Nicholas. I tried to cherry-pick to branch-2.6, however, it have several conflicts. Brahma Reddy Battula and Tsz Wo Nicholas Sze , would you help to merge the commit to branch-2.6 or deliver a 2.6 patch here? Thanks!
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          +1 we should as well cherry-pick this to branch-2.6.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - +1 we should as well cherry-pick this to branch-2.6.
          Hide
          djp Junping Du added a comment -

          Hi Tsz Wo Nicholas Sze and Brahma Reddy Battula, shall we cherry-pick this fix to branch-2.6 as well?

          Show
          djp Junping Du added a comment - Hi Tsz Wo Nicholas Sze and Brahma Reddy Battula , shall we cherry-pick this fix to branch-2.6 as well?
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Merged to 2.8. Thanks for the reminder.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Merged to 2.8. Thanks for the reminder.
          Hide
          kihwal Kihwal Lee added a comment -

          Please commit this to branch-2.8.

          Show
          kihwal Kihwal Lee added a comment - Please commit this to branch-2.8.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Tsz Wo Nicholas Sze thanks a lot for reviewing and committing this issue.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Tsz Wo Nicholas Sze thanks a lot for reviewing and committing this issue.
          Hide
          hudson Hudson added a comment -

          ABORTED: Integrated in Hadoop-Hdfs-trunk-Java8 #659 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/659/)
          HDFS-9294. DFSClient deadlock when close file and failed to renew lease. (szetszwo: rev e8bd1ba74b2fc7a6a1b71d068ef01a0fb0bbe294)

          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - ABORTED: Integrated in Hadoop-Hdfs-trunk-Java8 #659 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/659/ ) HDFS-9294 . DFSClient deadlock when close file and failed to renew lease. (szetszwo: rev e8bd1ba74b2fc7a6a1b71d068ef01a0fb0bbe294) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          I have committed this. Thanks, Brahma!

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Brahma!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8911 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8911/)
          HDFS-9294. DFSClient deadlock when close file and failed to renew lease. (szetszwo: rev e8bd1ba74b2fc7a6a1b71d068ef01a0fb0bbe294)

          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8911 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8911/ ) HDFS-9294 . DFSClient deadlock when close file and failed to renew lease. (szetszwo: rev e8bd1ba74b2fc7a6a1b71d068ef01a0fb0bbe294) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Thanks Chris Nauroth. This sounds not ideal since many useful hdfs tests are not executed when there is a hadoop-hdfs-client only change.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Thanks Chris Nauroth . This sounds not ideal since many useful hdfs tests are not executed when there is a hadoop-hdfs-client only change.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 9m 4s trunk passed
          +1 compile 0m 36s trunk passed with JDK v1.8.0_66
          +1 compile 0m 36s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 13s trunk passed
          +1 mvnsite 0m 41s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 2m 16s trunk passed
          +1 javadoc 0m 27s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 31s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 41s the patch passed
          +1 compile 0m 37s the patch passed with JDK v1.8.0_66
          +1 javac 0m 37s the patch passed
          +1 compile 0m 35s the patch passed with JDK v1.7.0_91
          +1 javac 0m 35s the patch passed
          +1 checkstyle 0m 13s the patch passed
          +1 mvnsite 0m 42s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 25s the patch passed
          +1 javadoc 0m 28s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 28s the patch passed with JDK v1.7.0_91
          +1 unit 1m 9s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
          +1 unit 1m 9s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 26s Patch does not generate ASF License warnings.
          26m 58s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775431/HDFS-9294-002.patch
          JIRA Issue HDFS-9294
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux cfdb1f784d72 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6b9a5be
          findbugs v3.0.0
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13744/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client
          Max memory used 76MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13744/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 9m 4s trunk passed +1 compile 0m 36s trunk passed with JDK v1.8.0_66 +1 compile 0m 36s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 13s trunk passed +1 mvnsite 0m 41s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 2m 16s trunk passed +1 javadoc 0m 27s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 31s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 41s the patch passed +1 compile 0m 37s the patch passed with JDK v1.8.0_66 +1 javac 0m 37s the patch passed +1 compile 0m 35s the patch passed with JDK v1.7.0_91 +1 javac 0m 35s the patch passed +1 checkstyle 0m 13s the patch passed +1 mvnsite 0m 42s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 25s the patch passed +1 javadoc 0m 28s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 28s the patch passed with JDK v1.7.0_91 +1 unit 1m 9s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. +1 unit 1m 9s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 26s Patch does not generate ASF License warnings. 26m 58s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775431/HDFS-9294-002.patch JIRA Issue HDFS-9294 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux cfdb1f784d72 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6b9a5be findbugs v3.0.0 JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13744/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13744/console This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          Hi Tsz Wo Nicholas Sze. That relates to the recent split of the client code into a separate hadoop-hdfs-client module. Tests are only executed in the modules being changed by a patch. This patch only changes files in hadoop-hdfs-client, not files in hadoop-hdfs.

          Show
          cnauroth Chris Nauroth added a comment - Hi Tsz Wo Nicholas Sze . That relates to the recent split of the client code into a separate hadoop-hdfs-client module. Tests are only executed in the modules being changed by a patch. This patch only changes files in hadoop-hdfs-client, not files in hadoop-hdfs.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Allen Wittenauer, why there were only 49 tests (-4077) executed in the previous build?

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Allen Wittenauer , why there were only 49 tests (-4077) executed in the previous build?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 8m 10s trunk passed
          +1 compile 0m 28s trunk passed with JDK v1.8.0_66
          +1 compile 0m 30s trunk passed with JDK v1.7.0_85
          +1 checkstyle 0m 12s trunk passed
          +1 mvnsite 0m 36s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 2m 3s trunk passed
          +1 javadoc 0m 21s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 25s trunk passed with JDK v1.7.0_85
          +1 mvninstall 0m 36s the patch passed
          +1 compile 0m 29s the patch passed with JDK v1.8.0_66
          +1 javac 0m 29s the patch passed
          +1 compile 0m 29s the patch passed with JDK v1.7.0_85
          +1 javac 0m 29s the patch passed
          +1 checkstyle 0m 10s the patch passed
          +1 mvnsite 0m 36s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 6s the patch passed
          +1 javadoc 0m 21s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 26s the patch passed with JDK v1.7.0_85
          +1 unit 0m 55s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
          +1 unit 1m 0s hadoop-hdfs-client in the patch passed with JDK v1.7.0_85.
          +1 asflicense 0m 24s Patch does not generate ASF License warnings.
          23m 29s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775431/HDFS-9294-002.patch
          JIRA Issue HDFS-9294
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 09ffb4893e70 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6b9a5be
          findbugs v3.0.0
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13742/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client
          Max memory used 76MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13742/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 10s trunk passed +1 compile 0m 28s trunk passed with JDK v1.8.0_66 +1 compile 0m 30s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 12s trunk passed +1 mvnsite 0m 36s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 2m 3s trunk passed +1 javadoc 0m 21s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 25s trunk passed with JDK v1.7.0_85 +1 mvninstall 0m 36s the patch passed +1 compile 0m 29s the patch passed with JDK v1.8.0_66 +1 javac 0m 29s the patch passed +1 compile 0m 29s the patch passed with JDK v1.7.0_85 +1 javac 0m 29s the patch passed +1 checkstyle 0m 10s the patch passed +1 mvnsite 0m 36s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 6s the patch passed +1 javadoc 0m 21s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 26s the patch passed with JDK v1.7.0_85 +1 unit 0m 55s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. +1 unit 1m 0s hadoop-hdfs-client in the patch passed with JDK v1.7.0_85. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 23m 29s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775431/HDFS-9294-002.patch JIRA Issue HDFS-9294 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 09ffb4893e70 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6b9a5be findbugs v3.0.0 JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13742/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13742/console This message was automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          +1 the new patch looks good.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - +1 the new patch looks good.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Reposting the trunk patch to trigger Jenkins.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Reposting the trunk patch to trigger Jenkins.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          -1 docker 0m 4s Docker failed to build yetus/hadoop:5d9212c.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775234/HDFS-9294-branch-2.patch
          JIRA Issue HDFS-9294
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13731/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 docker 0m 4s Docker failed to build yetus/hadoop:5d9212c. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12775234/HDFS-9294-branch-2.patch JIRA Issue HDFS-9294 Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13731/console This message was automatically generated.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Tsz Wo Nicholas Sze thanks a lot for your review..Yes, you are correct..Upload patches for trunk,branch-2 and branch-2.7

          Show
          brahmareddy Brahma Reddy Battula added a comment - Tsz Wo Nicholas Sze thanks a lot for your review..Yes, you are correct..Upload patches for trunk,branch-2 and branch-2.7
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 8m 57s trunk passed
          +1 compile 0m 35s trunk passed with JDK v1.8.0_66
          +1 compile 0m 33s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 14s trunk passed
          +1 mvnsite 0m 39s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 2m 9s trunk passed
          +1 javadoc 0m 26s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 27s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 39s the patch passed
          +1 compile 0m 34s the patch passed with JDK v1.8.0_66
          +1 javac 0m 34s the patch passed
          +1 compile 0m 34s the patch passed with JDK v1.7.0_91
          +1 javac 0m 34s the patch passed
          +1 checkstyle 0m 13s the patch passed
          +1 mvnsite 0m 40s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 18s the patch passed
          +1 javadoc 0m 26s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 28s the patch passed with JDK v1.7.0_91
          +1 unit 1m 2s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
          +1 unit 1m 4s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 27s Patch does not generate ASF License warnings.
          39m 22s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774976/HDFS-9294.patch
          JIRA Issue HDFS-9294
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 544e828443c9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 3c4a34e
          findbugs v3.0.0
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13724/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client
          Max memory used 76MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13724/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 57s trunk passed +1 compile 0m 35s trunk passed with JDK v1.8.0_66 +1 compile 0m 33s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 39s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 2m 9s trunk passed +1 javadoc 0m 26s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 27s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 39s the patch passed +1 compile 0m 34s the patch passed with JDK v1.8.0_66 +1 javac 0m 34s the patch passed +1 compile 0m 34s the patch passed with JDK v1.7.0_91 +1 javac 0m 34s the patch passed +1 checkstyle 0m 13s the patch passed +1 mvnsite 0m 40s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 18s the patch passed +1 javadoc 0m 26s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 28s the patch passed with JDK v1.7.0_91 +1 unit 1m 2s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. +1 unit 1m 4s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 27s Patch does not generate ASF License warnings. 39m 22s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774976/HDFS-9294.patch JIRA Issue HDFS-9294 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 544e828443c9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3c4a34e findbugs v3.0.0 JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13724/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13724/console This message was automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Also, let's move dfsClient.endFileLease(fileId); from closeImpl() to close(). Then, we can unsync close() and keep closeImpl() synchronized.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Also, let's move dfsClient.endFileLease(fileId); from closeImpl() to close(). Then, we can unsync close() and keep closeImpl() synchronized.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Thanks Brahma Reddy Battula. We also need a similar fix for DFSStripedOutputStream.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Thanks Brahma Reddy Battula . We also need a similar fix for DFSStripedOutputStream.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Had seen similar issue in our cluster as well.Attaching the patch, with solution mentioned by DENG FEI

          Show
          brahmareddy Brahma Reddy Battula added a comment - Had seen similar issue in our cluster as well.Attaching the patch, with solution mentioned by DENG FEI
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          DENG FEI, are you going to post a patch soon? We need this for 2.7.2.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - DENG FEI , are you going to post a patch soon? We need this for 2.7.2.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Vinod Kumar Vavilapalli, we probably should include this in 2.7.2.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Vinod Kumar Vavilapalli , we probably should include this in 2.7.2.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          > ... move "dfsClient.endFileLease" out of synchronized block ...

          It seems a good idea. Please feel free to post your patch. Thanks.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - > ... move "dfsClient.endFileLease" out of synchronized block ... It seems a good idea. Please feel free to post your patch. Thanks.
          Hide
          Deng FEI DENG FEI added a comment -

          dfsClient.endFileLease seem idempotent,and not have side-effect.

          Show
          Deng FEI DENG FEI added a comment - dfsClient.endFileLease seem idempotent,and not have side-effect.
          Hide
          Deng FEI DENG FEI added a comment -

          Sorry,It's too late finish it.
          This is our solution,we just move "dfsClient.endFileLease" out of synchronized block which try to acquire LeaseRenewer lock while DFSOutputStream is closing or aborting, rather then set 'closing' or 'aborting' flag on DFSOutputStream,and let the LeaseRenewer thread detect and skip.
          but is't not graceful,do you have any better idea?

          Show
          Deng FEI DENG FEI added a comment - Sorry,It's too late finish it. This is our solution,we just move "dfsClient.endFileLease" out of synchronized block which try to acquire LeaseRenewer lock while DFSOutputStream is closing or aborting, rather then set 'closing' or 'aborting' flag on DFSOutputStream,and let the LeaseRenewer thread detect and skip. but is't not graceful,do you have any better idea?
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          > ... and it seems not solved at Hadoop 2.7.1 .If confirmed , we can fixed the issue.

          I checked the code. Trunk has this bug. I guess Hadoop 2.7.1 should also has this bug.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - > ... and it seems not solved at Hadoop 2.7.1 .If confirmed , we can fixed the issue. I checked the code. Trunk has this bug. I guess Hadoop 2.7.1 should also has this bug.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Thanks for reporting the bug. Are you working on a fix?

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Thanks for reporting the bug. Are you working on a fix?

            People

            • Assignee:
              brahmareddy Brahma Reddy Battula
              Reporter:
              Deng FEI DENG FEI
            • Votes:
              0 Vote for this issue
              Watchers:
              20 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development