Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9412

getBlocks occupies FSLock and takes too long to complete

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      getBlocks in NameNodeRpcServer acquires a read lock then may take a long time to complete (probably several seconds, if number of blocks are too much).
      During this period, other threads attempting to acquire write lock will wait.
      In an extreme case, RPC handlers are occupied by one reader thread calling getBlocks and all other threads waiting for write lock, rpc server acts like hung. Unfortunately, this tends to happen in heavy loaded cluster, since read operations come and go fast (they do not need to wait), leaving write operations waiting.

      Looks like we can optimize this thing like DN block report did in past, by splitting the operation into smaller sub operations, and let other threads do their work between each sub operation. The whole result is returned at once, though (one thing different from DN block report).
      I am not sure whether this will work. Any better idea?

      1. HDFS-9412.0000.patch
        2 kB
        He Tianyi
      2. HDFS-9412.0001.patch
        2 kB
        He Tianyi
      3. HDFS-9412.0002.patch
        4 kB
        He Tianyi

        Issue Links

          Activity

          Hide
          He Tianyi He Tianyi added a comment -

          Sorry the misuse of word "starvation". It's not starvation exactly, just hung for a while.

          Show
          He Tianyi He Tianyi added a comment - Sorry the misuse of word "starvation". It's not starvation exactly, just hung for a while.
          Hide
          He Tianyi He Tianyi added a comment -

          A simpler approach would be ignore small blocks during iteration (just as Balancer did in HDFS-8824, if HDFS-8824 is implemented, Balancer don't need these blocks anyway), this will reduce the cost of gathering enough amount of size of data (I did not do the profiling but perhaps there are enough overhead in addBlock, lots of objects are created).

          Show
          He Tianyi He Tianyi added a comment - A simpler approach would be ignore small blocks during iteration (just as Balancer did in HDFS-8824 , if HDFS-8824 is implemented, Balancer don't need these blocks anyway), this will reduce the cost of gathering enough amount of size of data (I did not do the profiling but perhaps there are enough overhead in addBlock , lots of objects are created).
          Hide
          andrew.wang Andrew Wang added a comment -

          If the issue is a writer blocking the RWLock, you can turn on unfair locking for FSNamesystem: dfs.namenode.fslock.fair to false. This will at least allow other readers through, which will help throughput.

          Show
          andrew.wang Andrew Wang added a comment - If the issue is a writer blocking the RWLock, you can turn on unfair locking for FSNamesystem: dfs.namenode.fslock.fair to false . This will at least allow other readers through, which will help throughput.
          Hide
          andrew.wang Andrew Wang added a comment -

          Also many of our listing operations return RemoteIterators which query only small batches at a time. Is it possible your application isn't using the RemoteIterator APIs?

          Show
          andrew.wang Andrew Wang added a comment - Also many of our listing operations return RemoteIterators which query only small batches at a time. Is it possible your application isn't using the RemoteIterator APIs?
          Hide
          He Tianyi He Tianyi added a comment -

          Andrew Wang Perhaps switching to unfair RWLock may cause other issues, since machine running NameNode does not necessarily have SMP architecture.

          I think this is due to having many small blocks in cluster, getBlocks is called by Balancer and will not return until exhausted or total size satisfies, and there are actually many threads doing the same thing (dfs.balancer.dispatcherThreads).
          Besides decreasing number of threads, maybe we can make this faster either.

          Show
          He Tianyi He Tianyi added a comment - Andrew Wang Perhaps switching to unfair RWLock may cause other issues, since machine running NameNode does not necessarily have SMP architecture. I think this is due to having many small blocks in cluster, getBlocks is called by Balancer and will not return until exhausted or total size satisfies, and there are actually many threads doing the same thing ( dfs.balancer.dispatcherThreads ). Besides decreasing number of threads, maybe we can make this faster either.
          Hide
          He Tianyi He Tianyi added a comment -

          A patch that skip small blocks in getBlocks. These are unused anyway.
          This reduces average queue time of NameNode nearly 40% shorter during burst getBlocks requests with 160 dispatcher threads (in my cluster).

          Show
          He Tianyi He Tianyi added a comment - A patch that skip small blocks in getBlocks . These are unused anyway. This reduces average queue time of NameNode nearly 40% shorter during burst getBlocks requests with 160 dispatcher threads (in my cluster).
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 8m 7s trunk passed
          +1 compile 0m 44s trunk passed with JDK v1.8.0_66
          +1 compile 0m 43s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 17s trunk passed
          +1 mvnsite 0m 54s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 59s trunk passed
          +1 javadoc 1m 11s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 50s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 51s the patch passed
          +1 compile 0m 44s the patch passed with JDK v1.8.0_66
          -1 javac 6m 35s hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66 with JDK v1.8.0_66 generated 1 new issues (was 32, now 32).
          +1 javac 0m 44s the patch passed
          +1 compile 0m 44s the patch passed with JDK v1.7.0_91
          -1 javac 7m 19s hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91 with JDK v1.7.0_91 generated 1 new issues (was 34, now 34).
          +1 javac 0m 44s the patch passed
          -1 checkstyle 0m 17s Patch generated 1 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 161, now 161).
          +1 mvnsite 0m 53s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 8s the patch passed
          +1 javadoc 1m 9s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 51s the patch passed with JDK v1.7.0_91
          -1 unit 56m 15s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 54m 1s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          +1 asflicense 0m 27s Patch does not generate ASF License warnings.
          138m 21s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMetrics
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.TestGetBlocks
            hadoop.hdfs.server.namenode.TestFSImageWithAcl
            hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.TestModTime
            hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
            hadoop.hdfs.server.datanode.TestBlockReplacement
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140
            hadoop.hdfs.TestGetBlocks
            hadoop.hdfs.TestSeekBug
            hadoop.hdfs.server.namenode.TestNNThroughputBenchmark



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12776303/HDFS-9412.0000.patch
          JIRA Issue HDFS-9412
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 4720e950b81d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 52ad912
          findbugs v3.0.0
          javac hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13949/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          javac hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HDFS-Build/13949/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13949/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13949/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13949/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13949/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13949/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13949/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 75MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13949/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 7s trunk passed +1 compile 0m 44s trunk passed with JDK v1.8.0_66 +1 compile 0m 43s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 54s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 59s trunk passed +1 javadoc 1m 11s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 50s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 51s the patch passed +1 compile 0m 44s the patch passed with JDK v1.8.0_66 -1 javac 6m 35s hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66 with JDK v1.8.0_66 generated 1 new issues (was 32, now 32). +1 javac 0m 44s the patch passed +1 compile 0m 44s the patch passed with JDK v1.7.0_91 -1 javac 7m 19s hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91 with JDK v1.7.0_91 generated 1 new issues (was 34, now 34). +1 javac 0m 44s the patch passed -1 checkstyle 0m 17s Patch generated 1 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 161, now 161). +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 8s the patch passed +1 javadoc 1m 9s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 51s the patch passed with JDK v1.7.0_91 -1 unit 56m 15s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 54m 1s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 27s Patch does not generate ASF License warnings. 138m 21s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.TestGetBlocks   hadoop.hdfs.server.namenode.TestFSImageWithAcl   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.TestModTime   hadoop.hdfs.server.namenode.TestNNThroughputBenchmark   hadoop.hdfs.server.datanode.TestBlockReplacement JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140   hadoop.hdfs.TestGetBlocks   hadoop.hdfs.TestSeekBug   hadoop.hdfs.server.namenode.TestNNThroughputBenchmark Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12776303/HDFS-9412.0000.patch JIRA Issue HDFS-9412 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 4720e950b81d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 52ad912 findbugs v3.0.0 javac hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13949/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt javac hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HDFS-Build/13949/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13949/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13949/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13949/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13949/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13949/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13949/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 75MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13949/console This message was automatically generated.
          Hide
          He Tianyi He Tianyi added a comment -

          Added ~200 nodes to HDFS cluster recently. As balancing become radical, NameNode still have spikes in latency, but better than before (usually peak queue time reaches 10~20s, now is 5~6s).

          Show
          He Tianyi He Tianyi added a comment - Added ~200 nodes to HDFS cluster recently. As balancing become radical, NameNode still have spikes in latency, but better than before (usually peak queue time reaches 10~20s, now is 5~6s).
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 4s HDFS-9412 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12776303/HDFS-9412.0000.patch
          JIRA Issue HDFS-9412
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15147/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 4s HDFS-9412 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12776303/HDFS-9412.0000.patch JIRA Issue HDFS-9412 Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15147/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          walter.k.su Walter Su added a comment -

          One thread holding a readLock too long is very like holding a writeLock. We should avoid that. And after HDFS-8824, the small blocks are unused anyway, so there's no point to send them to balancer.
          Hi, He Tianyi, Do you mind rebase the patch?

          Show
          walter.k.su Walter Su added a comment - One thread holding a readLock too long is very like holding a writeLock. We should avoid that. And after HDFS-8824 , the small blocks are unused anyway, so there's no point to send them to balancer. Hi, He Tianyi , Do you mind rebase the patch?
          Hide
          He Tianyi He Tianyi added a comment -

          Rebase against current trunk.

          Show
          He Tianyi He Tianyi added a comment - Rebase against current trunk.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 41s trunk passed
          +1 compile 0m 55s trunk passed with JDK v1.8.0_77
          +1 compile 0m 47s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 22s trunk passed
          +1 mvnsite 1m 9s trunk passed
          +1 mvneclipse 0m 21s trunk passed
          +1 findbugs 2m 34s trunk passed
          +1 javadoc 1m 47s trunk passed with JDK v1.8.0_77
          +1 javadoc 2m 28s trunk passed with JDK v1.7.0_95
          +1 mvninstall 1m 2s the patch passed
          +1 compile 1m 4s the patch passed with JDK v1.8.0_77
          +1 javac 1m 4s the patch passed
          +1 compile 0m 52s the patch passed with JDK v1.7.0_95
          +1 javac 0m 52s the patch passed
          -1 checkstyle 0m 29s hadoop-hdfs-project/hadoop-hdfs: patch generated 1 new + 130 unchanged - 0 fixed = 131 total (was 130)
          +1 mvnsite 1m 14s the patch passed
          +1 mvneclipse 0m 19s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 findbugs 2m 59s the patch passed
          +1 javadoc 1m 48s the patch passed with JDK v1.8.0_77
          +1 javadoc 2m 11s the patch passed with JDK v1.7.0_95
          -1 unit 115m 4s hadoop-hdfs in the patch failed with JDK v1.8.0_77.
          -1 unit 107m 12s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 31s Patch does not generate ASF License warnings.
          256m 9s



          Reason Tests
          JDK v1.8.0_77 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeUUID
            hadoop.hdfs.TestBlockStoragePolicy
            hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
            hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages
            hadoop.hdfs.TestReadStripedFileWithMissingBlocks
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.server.datanode.TestDiskError
            hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
            hadoop.hdfs.TestGetBlocks
            hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
            hadoop.hdfs.server.datanode.TestDataNodeLifeline
            hadoop.hdfs.server.datanode.TestIncrementalBrVariations
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage
            hadoop.hdfs.server.namenode.TestEditLog
            hadoop.hdfs.server.datanode.TestCachingStrategy
            hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
            hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
            hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider
            hadoop.hdfs.server.blockmanagement.TestNodeCount
            hadoop.hdfs.server.namenode.TestCacheDirectives
            hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork
            hadoop.hdfs.server.datanode.TestDirectoryScanner
          JDK v1.8.0_77 Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile
            org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeUUID
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
            hadoop.hdfs.TestReadStripedFileWithMissingBlocks
            hadoop.hdfs.server.namenode.TestNamenodeRetryCache
            hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.server.namenode.TestParallelImageWrite
            hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.TestGetBlocks
            hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.server.namenode.snapshot.TestSnapshot
            hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica
            hadoop.hdfs.server.datanode.TestDirectoryScanner
          JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile
            org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12798500/HDFS-9412.0001.patch
          JIRA Issue HDFS-9412
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux c1d54459586d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 903428b
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15148/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/15148/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15148/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15148/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15148/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15148/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15148/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15148/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 41s trunk passed +1 compile 0m 55s trunk passed with JDK v1.8.0_77 +1 compile 0m 47s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 22s trunk passed +1 mvnsite 1m 9s trunk passed +1 mvneclipse 0m 21s trunk passed +1 findbugs 2m 34s trunk passed +1 javadoc 1m 47s trunk passed with JDK v1.8.0_77 +1 javadoc 2m 28s trunk passed with JDK v1.7.0_95 +1 mvninstall 1m 2s the patch passed +1 compile 1m 4s the patch passed with JDK v1.8.0_77 +1 javac 1m 4s the patch passed +1 compile 0m 52s the patch passed with JDK v1.7.0_95 +1 javac 0m 52s the patch passed -1 checkstyle 0m 29s hadoop-hdfs-project/hadoop-hdfs: patch generated 1 new + 130 unchanged - 0 fixed = 131 total (was 130) +1 mvnsite 1m 14s the patch passed +1 mvneclipse 0m 19s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 2m 59s the patch passed +1 javadoc 1m 48s the patch passed with JDK v1.8.0_77 +1 javadoc 2m 11s the patch passed with JDK v1.7.0_95 -1 unit 115m 4s hadoop-hdfs in the patch failed with JDK v1.8.0_77. -1 unit 107m 12s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 31s Patch does not generate ASF License warnings. 256m 9s Reason Tests JDK v1.8.0_77 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeUUID   hadoop.hdfs.TestBlockStoragePolicy   hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages   hadoop.hdfs.TestReadStripedFileWithMissingBlocks   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.server.datanode.TestDiskError   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.TestGetBlocks   hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead   hadoop.hdfs.server.datanode.TestDataNodeLifeline   hadoop.hdfs.server.datanode.TestIncrementalBrVariations   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage   hadoop.hdfs.server.namenode.TestEditLog   hadoop.hdfs.server.datanode.TestCachingStrategy   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport   hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks   hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider   hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.server.namenode.TestCacheDirectives   hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork   hadoop.hdfs.server.datanode.TestDirectoryScanner JDK v1.8.0_77 Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile   org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeUUID   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.TestReadStripedFileWithMissingBlocks   hadoop.hdfs.server.namenode.TestNamenodeRetryCache   hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.server.namenode.TestParallelImageWrite   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.TestGetBlocks   hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.namenode.snapshot.TestSnapshot   hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica   hadoop.hdfs.server.datanode.TestDirectoryScanner JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile   org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12798500/HDFS-9412.0001.patch JIRA Issue HDFS-9412 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c1d54459586d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 903428b Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15148/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/15148/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15148/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15148/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15148/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15148/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15148/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15148/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          walter.k.su Walter Su added a comment -

          Thank you for updating. The test TestGetBlocks failed. Do you mind changing the test accordingly? And fix the checkstyle issue as well.

          Show
          walter.k.su Walter Su added a comment - Thank you for updating. The test TestGetBlocks failed. Do you mind changing the test accordingly? And fix the checkstyle issue as well.
          Hide
          He Tianyi He Tianyi added a comment -

          Fix codestyle, whitespace and unit test.

          Show
          He Tianyi He Tianyi added a comment - Fix codestyle, whitespace and unit test.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 11m 29s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 44s trunk passed
          +1 compile 0m 39s trunk passed with JDK v1.8.0_77
          +1 compile 0m 41s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 22s trunk passed
          +1 mvnsite 0m 52s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 56s trunk passed
          +1 javadoc 1m 4s trunk passed with JDK v1.8.0_77
          +1 javadoc 1m 47s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 45s the patch passed
          +1 compile 0m 38s the patch passed with JDK v1.8.0_77
          +1 javac 0m 38s the patch passed
          +1 compile 0m 38s the patch passed with JDK v1.7.0_95
          +1 javac 0m 38s the patch passed
          +1 checkstyle 0m 21s the patch passed
          +1 mvnsite 0m 47s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 9s the patch passed
          +1 javadoc 1m 3s the patch passed with JDK v1.8.0_77
          +1 javadoc 1m 45s the patch passed with JDK v1.7.0_95
          -1 unit 72m 0s hadoop-hdfs in the patch failed with JDK v1.8.0_77.
          -1 unit 65m 40s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 27s Patch does not generate ASF License warnings.
          174m 30s



          Reason Tests
          JDK v1.8.0_77 Failed junit tests hadoop.hdfs.tools.TestDFSAdmin
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
            hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
            hadoop.hdfs.TestFileAppend
            hadoop.hdfs.TestReadStripedFileWithMissingBlocks
          JDK v1.8.0_77 Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile
            org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestHFlush
            hadoop.hdfs.tools.TestDFSAdmin
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
            hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
            hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration
            hadoop.hdfs.TestReadStripedFileWithMissingBlocks
            hadoop.hdfs.server.balancer.TestBalancer
            hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
          JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile
            org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12798643/HDFS-9412.0002.patch
          JIRA Issue HDFS-9412
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux dd56560770dd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 27b131e
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15159/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15159/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15159/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15159/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15159/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15159/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 11m 29s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 44s trunk passed +1 compile 0m 39s trunk passed with JDK v1.8.0_77 +1 compile 0m 41s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 22s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 56s trunk passed +1 javadoc 1m 4s trunk passed with JDK v1.8.0_77 +1 javadoc 1m 47s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 45s the patch passed +1 compile 0m 38s the patch passed with JDK v1.8.0_77 +1 javac 0m 38s the patch passed +1 compile 0m 38s the patch passed with JDK v1.7.0_95 +1 javac 0m 38s the patch passed +1 checkstyle 0m 21s the patch passed +1 mvnsite 0m 47s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 9s the patch passed +1 javadoc 1m 3s the patch passed with JDK v1.8.0_77 +1 javadoc 1m 45s the patch passed with JDK v1.7.0_95 -1 unit 72m 0s hadoop-hdfs in the patch failed with JDK v1.8.0_77. -1 unit 65m 40s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 27s Patch does not generate ASF License warnings. 174m 30s Reason Tests JDK v1.8.0_77 Failed junit tests hadoop.hdfs.tools.TestDFSAdmin   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead   hadoop.hdfs.TestFileAppend   hadoop.hdfs.TestReadStripedFileWithMissingBlocks JDK v1.8.0_77 Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile   org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestHFlush   hadoop.hdfs.tools.TestDFSAdmin   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks   hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration   hadoop.hdfs.TestReadStripedFileWithMissingBlocks   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile   org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12798643/HDFS-9412.0002.patch JIRA Issue HDFS-9412 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux dd56560770dd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 27b131e Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/15159/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15159/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15159/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15159/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15159/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15159/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          walter.k.su Walter Su added a comment -

          TestBalancer passes locally. +1 for the last patch.

          Show
          walter.k.su Walter Su added a comment - TestBalancer passes locally. +1 for the last patch.
          Hide
          He Tianyi He Tianyi added a comment -

          Hi, Walter Su. Would you commit this patch when available?

          Show
          He Tianyi He Tianyi added a comment - Hi, Walter Su . Would you commit this patch when available?
          Hide
          walter.k.su Walter Su added a comment -

          Committed to trunk, branch-2, branch-2.8. Thanks He Tianyi for contribution!

          Show
          walter.k.su Walter Su added a comment - Committed to trunk, branch-2, branch-2.8. Thanks He Tianyi for contribution!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9625 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9625/)
          HDFS-9412. getBlocks occupies FSLock and takes too long to complete. (waltersu4549: rev 67523ffcf491f4f2db5335899c00a174d0caaa9b)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestGetBlocks.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9625 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9625/ ) HDFS-9412 . getBlocks occupies FSLock and takes too long to complete. (waltersu4549: rev 67523ffcf491f4f2db5335899c00a174d0caaa9b) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestGetBlocks.java

            People

            • Assignee:
              He Tianyi He Tianyi
              Reporter:
              He Tianyi He Tianyi
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development