Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8674

Improve performance of postponed block scans

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha2
    • Component/s: namenode
    • Labels:
      None

      Description

      When a standby goes active, it marks all nodes as "stale" which will cause block invalidations for over-replicated blocks to be queued until full block reports are received from the nodes with the block. The replication monitor scans the queue with O(N) runtime. It picks a random offset and iterates through the set to randomize blocks scanned.

      The result is devastating when a cluster loses multiple nodes during a rolling upgrade. Re-replication occurs, the nodes come back, the excess block invalidations are postponed. Rescanning just 2k blocks out of millions of postponed blocks may take multiple seconds. During the scan, the write lock is held which stalls all other processing.

      1. HDFS-8674.2.patch
        7 kB
        Daryn Sharp
      2. HDFS-8674.branch-2.patch
        7 kB
        Daryn Sharp
      3. HDFS-8674.patch
        6 kB
        Daryn Sharp
      4. HDFS-8674.patch
        6 kB
        Daryn Sharp
      5. HDFS-8674.trunk.2.patch
        8 kB
        Daryn Sharp
      6. HDFS-8674.trunk.patch
        8 kB
        Daryn Sharp

        Issue Links

          Activity

          Hide
          daryn Daryn Sharp added a comment -

          No additional tests due to simple performance optimization that existing tests cover.

          Show
          daryn Daryn Sharp added a comment - No additional tests due to simple performance optimization that existing tests cover.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 15m 16s Findbugs (version ) appears to be broken on trunk.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 30s There were no new javac warning messages.
          +1 javadoc 9m 38s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 50s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 35s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          -1 findbugs 3m 24s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
          +1 native 3m 14s Pre-build of native portion
          +1 hdfs tests 162m 31s Tests passed in hadoop-hdfs.
              204m 57s  



          Reason Tests
          FindBugs module:hadoop-hdfs



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12742164/HDFS-8674.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 60b858b
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/11502/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11502/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11502/testReport/
          Java 1.7.0_55
          uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11502/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 15m 16s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 30s There were no new javac warning messages. +1 javadoc 9m 38s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 50s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. -1 findbugs 3m 24s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 native 3m 14s Pre-build of native portion +1 hdfs tests 162m 31s Tests passed in hadoop-hdfs.     204m 57s   Reason Tests FindBugs module:hadoop-hdfs Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12742164/HDFS-8674.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 60b858b Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/11502/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11502/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11502/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11502/console This message was automatically generated.
          Hide
          mingma Ming Ma added a comment -

          Thanks Daryn Sharp!

          • The motivation of randomization is to make sure we don't end up scanning the same blocks if there isn't much update to postponedMisreplicatedBlocks. But maybe that doesn't matter much as it will eventually be drained; also HashSet doesn't make order guarantee.
          • Does the latency of couple seconds come from HashSet iteration? Some quick test indicates iterating through 5M entries of a HashSet take around 50ms.
          • What is the reason to change from HashSet to LinkedHashSet, to have some order guarantee? HashSet is faster than LinkedHashSet.
          • During the scan, if a block is marked as POSTPONE, it will be removed from postponedMisreplicatedBlocks first and add it back later via rescannedMisreplicatedBlocks. Can it just remove those blocks from postponedMisreplicatedBlocks that aren't marked as POSTPONE without using rescannedMisreplicatedBlocks?
          Show
          mingma Ming Ma added a comment - Thanks Daryn Sharp ! The motivation of randomization is to make sure we don't end up scanning the same blocks if there isn't much update to postponedMisreplicatedBlocks. But maybe that doesn't matter much as it will eventually be drained; also HashSet doesn't make order guarantee. Does the latency of couple seconds come from HashSet iteration? Some quick test indicates iterating through 5M entries of a HashSet take around 50ms. What is the reason to change from HashSet to LinkedHashSet, to have some order guarantee? HashSet is faster than LinkedHashSet. During the scan, if a block is marked as POSTPONE, it will be removed from postponedMisreplicatedBlocks first and add it back later via rescannedMisreplicatedBlocks. Can it just remove those blocks from postponedMisreplicatedBlocks that aren't marked as POSTPONE without using rescannedMisreplicatedBlocks?
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Daryn Sharp / Ming Ma, any update on this? Considering this for a 2.7.2 RC this weekend. Thanks.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Daryn Sharp / Ming Ma , any update on this? Considering this for a 2.7.2 RC this weekend. Thanks.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 0s The patch command could not apply the patch during dryrun.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12742164/HDFS-8674.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 96677be
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13211/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12742164/HDFS-8674.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 96677be Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13211/console This message was automatically generated.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Moving this out of 2.7.2 as there's been no update in a while.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Moving this out of 2.7.2 as there's been no update in a while.
          Hide
          daryn Daryn Sharp added a comment -

          Ming Ma It's a LinkedHashSet to cover the cases you state.

          The motivation of randomization is to make sure we don't end up scanning the same blocks if there isn't much update to postponedMisreplicatedBlocks. <...> What is the reason to change from HashSet to LinkedHashSet, to have some order guarantee? HashSet is faster than LinkedHashSet. <...> During the scan, if a block is marked as POSTPONE, it will be removed from postponedMisreplicatedBlocks first and add it back later via rescannedMisreplicatedBlocks. Can it just remove those blocks from postponedMisreplicatedBlocks that aren't marked as POSTPONE without using rescannedMisreplicatedBlocks?

          The LHS iterator is effectively functioning as a cheap circular array with acceptable insert/remove performance. Yes, the order guarantee ensures all blocks are visited w/o "skipping" to a random location. The block always has to be removed and re-inserted for the ordering to work. Otherwise the same blocks will be scanned over and over again.

          Does the latency of couple seconds come from HashSet iteration? Some quick test indicates iterating through 5M entries of a HashSet take around 50ms.

          Yes, the cycles wasted to skip the through the set were the killer. A micro-benchmark doesn't capture the same chaotic runtime environment of over a 100 threads competing for resources. Kihwal says (ironically) the performance "improved" under 5M.

          In multiple production incidents, the postponed queue backed up with 10-20M+ blocks. Scans determined the blocks are all still postponed. Pre-patch, the cycles took up to a few seconds and averaged in the many hundreds of ms. Performance was obviously atrocious. Post-patch, same scans took no more than a few ms.

          Show
          daryn Daryn Sharp added a comment - Ming Ma It's a LinkedHashSet to cover the cases you state. The motivation of randomization is to make sure we don't end up scanning the same blocks if there isn't much update to postponedMisreplicatedBlocks. <...> What is the reason to change from HashSet to LinkedHashSet, to have some order guarantee? HashSet is faster than LinkedHashSet. <...> During the scan, if a block is marked as POSTPONE, it will be removed from postponedMisreplicatedBlocks first and add it back later via rescannedMisreplicatedBlocks. Can it just remove those blocks from postponedMisreplicatedBlocks that aren't marked as POSTPONE without using rescannedMisreplicatedBlocks? The LHS iterator is effectively functioning as a cheap circular array with acceptable insert/remove performance. Yes, the order guarantee ensures all blocks are visited w/o "skipping" to a random location. The block always has to be removed and re-inserted for the ordering to work. Otherwise the same blocks will be scanned over and over again. Does the latency of couple seconds come from HashSet iteration? Some quick test indicates iterating through 5M entries of a HashSet take around 50ms. Yes, the cycles wasted to skip the through the set were the killer. A micro-benchmark doesn't capture the same chaotic runtime environment of over a 100 threads competing for resources. Kihwal says (ironically) the performance "improved" under 5M. In multiple production incidents, the postponed queue backed up with 10-20M+ blocks. Scans determined the blocks are all still postponed. Pre-patch, the cycles took up to a few seconds and averaged in the many hundreds of ms. Performance was obviously atrocious. Post-patch, same scans took no more than a few ms.
          Hide
          daryn Daryn Sharp added a comment -

          Ming Ma, comments?

          Show
          daryn Daryn Sharp added a comment - Ming Ma , comments?
          Hide
          mingma Ming Ma added a comment -

          Thanks Daryn Sharp. Your explanation makes sense. It is good to know the perf difference between "the # of items < 5M" and "larger number of items". +1 on the patch.

          Show
          mingma Ming Ma added a comment - Thanks Daryn Sharp . Your explanation makes sense. It is good to know the perf difference between "the # of items < 5M" and "larger number of items". +1 on the patch.
          Hide
          daryn Daryn Sharp added a comment -

          Updated patch since a data structure change prevented it from applying cleanly.

          Show
          daryn Daryn Sharp added a comment - Updated patch since a data structure change prevented it from applying cleanly.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 30s trunk passed
          +1 compile 0m 40s trunk passed with JDK v1.8.0_66
          +1 compile 0m 41s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 15s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 52s trunk passed
          +1 javadoc 1m 5s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 52s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 48s the patch passed
          +1 compile 0m 38s the patch passed with JDK v1.8.0_66
          -1 javac 6m 16s hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66 with JDK v1.8.0_66 generated 1 new issues (was 32, now 32).
          +1 javac 0m 38s the patch passed
          +1 compile 0m 40s the patch passed with JDK v1.7.0_91
          -1 javac 6m 57s hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91 with JDK v1.7.0_91 generated 1 new issues (was 34, now 34).
          +1 javac 0m 40s the patch passed
          -1 checkstyle 0m 14s Patch generated 1 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 161, now 160).
          +1 mvnsite 0m 50s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 2s the patch passed
          +1 javadoc 1m 4s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 46s the patch passed with JDK v1.7.0_91
          -1 unit 52m 16s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 50m 8s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 25s Patch generated 58 ASF License warnings.
          128m 44s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestDFSClientRetries
            hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication
            hadoop.hdfs.server.namenode.TestDecommissioningStatus
            hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
            hadoop.hdfs.server.namenode.ha.TestDNFencing
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050
            hadoop.hdfs.TestDFSUpgradeFromImage
            hadoop.hdfs.TestSetrepDecreasing
            hadoop.hdfs.server.blockmanagement.TestNodeCount
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
            hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication
            hadoop.hdfs.TestLeaseRecovery2
            hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
            hadoop.hdfs.server.namenode.ha.TestDNFencing
            hadoop.hdfs.TestSetrepDecreasing
            hadoop.hdfs.server.blockmanagement.TestNodeCount
            hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778098/HDFS-8674.patch
          JIRA Issue HDFS-8674
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 4f4d2df9b67c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0f708d4
          findbugs v3.0.0
          javac hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13903/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          javac hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HDFS-Build/13903/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13903/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13903/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13903/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13903/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13903/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13903/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13903/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 75MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13903/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 30s trunk passed +1 compile 0m 40s trunk passed with JDK v1.8.0_66 +1 compile 0m 41s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 52s trunk passed +1 javadoc 1m 5s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 52s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 48s the patch passed +1 compile 0m 38s the patch passed with JDK v1.8.0_66 -1 javac 6m 16s hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66 with JDK v1.8.0_66 generated 1 new issues (was 32, now 32). +1 javac 0m 38s the patch passed +1 compile 0m 40s the patch passed with JDK v1.7.0_91 -1 javac 6m 57s hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91 with JDK v1.7.0_91 generated 1 new issues (was 34, now 34). +1 javac 0m 40s the patch passed -1 checkstyle 0m 14s Patch generated 1 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 161, now 160). +1 mvnsite 0m 50s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 2s the patch passed +1 javadoc 1m 4s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 46s the patch passed with JDK v1.7.0_91 -1 unit 52m 16s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 50m 8s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 25s Patch generated 58 ASF License warnings. 128m 44s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestDFSClientRetries   hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes   hadoop.hdfs.server.namenode.ha.TestDNFencing   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050   hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.TestSetrepDecreasing   hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication   hadoop.hdfs.TestLeaseRecovery2   hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes   hadoop.hdfs.server.namenode.ha.TestDNFencing   hadoop.hdfs.TestSetrepDecreasing   hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778098/HDFS-8674.patch JIRA Issue HDFS-8674 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 4f4d2df9b67c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0f708d4 findbugs v3.0.0 javac hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13903/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt javac hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HDFS-Build/13903/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13903/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13903/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13903/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13903/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13903/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13903/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13903/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 75MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13903/console This message was automatically generated.
          Hide
          mingma Ming Ma added a comment -

          Daryn, from failed tests's logs, the latest patch uses LightWeightLinkedSet whose iterator doesn't support remove.

          Show
          mingma Ming Ma added a comment - Daryn, from failed tests's logs, the latest patch uses LightWeightLinkedSet whose iterator doesn't support remove.
          Hide
          daryn Daryn Sharp added a comment -

          I'll try to rebase again.

          Show
          daryn Daryn Sharp added a comment - I'll try to rebase again.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Daryn Sharp / Ming Ma, is it possible to get this in 2.7.3 in a week? Tx?

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Daryn Sharp / Ming Ma , is it possible to get this in 2.7.3 in a week? Tx?
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Haven't gotten any update, dropping this off into 2.7.4..

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Haven't gotten any update, dropping this off into 2.7.4..
          Hide
          andrew.wang Andrew Wang added a comment -

          Added some more target versions for tracking. Seems like this would still be good to get in.

          Show
          andrew.wang Andrew Wang added a comment - Added some more target versions for tracking. Seems like this would still be good to get in.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 6s HDFS-8674 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Issue HDFS-8674
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778098/HDFS-8674.patch
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17194/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 6s HDFS-8674 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HDFS-8674 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778098/HDFS-8674.patch Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17194/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          daryn Daryn Sharp added a comment -

          I'll update this patch.

          Show
          daryn Daryn Sharp added a comment - I'll update this patch.
          Hide
          daryn Daryn Sharp added a comment -

          Resolved minor conflicts.

          Show
          daryn Daryn Sharp added a comment - Resolved minor conflicts.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 9s HDFS-8674 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Issue HDFS-8674
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840925/HDFS-8674.2.patch
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17702/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 9s HDFS-8674 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HDFS-8674 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840925/HDFS-8674.2.patch Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17702/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          kihwal Kihwal Lee added a comment -

          Looks like HDFS-8674.2.patch is for branch-2. Can you post a trunk version?

          Show
          kihwal Kihwal Lee added a comment - Looks like HDFS-8674 .2.patch is for branch-2. Can you post a trunk version?
          Hide
          daryn Daryn Sharp added a comment -

          fix conflicts on trunk stemming from under replicated now being low redundancy.

          Show
          daryn Daryn Sharp added a comment - fix conflicts on trunk stemming from under replicated now being low redundancy.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 15m 46s trunk passed
          +1 compile 2m 5s trunk passed
          +1 checkstyle 1m 23s trunk passed
          +1 mvnsite 1m 21s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          -1 findbugs 1m 49s hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant Findbugs warnings.
          +1 javadoc 0m 41s trunk passed
          +1 mvninstall 0m 49s the patch passed
          +1 compile 0m 49s the patch passed
          +1 javac 0m 49s the patch passed
          +1 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 119 unchanged - 2 fixed = 119 total (was 121)
          +1 mvnsite 0m 54s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 56s the patch passed
          +1 javadoc 0m 39s the patch passed
          -1 unit 74m 34s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          105m 37s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.blockmanagement.TestNodeCount
            hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
            hadoop.hdfs.server.datanode.TestDataNodeUUID
            hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
            hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness
            hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
            hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker
            hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
            hadoop.hdfs.server.datanode.TestDataNodeMXBean
            hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR
            hadoop.hdfs.TestLeaseRecoveryStriped
            hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.hdfs.server.namenode.snapshot.TestSnapshot
            hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication
            hadoop.hdfs.server.namenode.ha.TestDNFencing



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-8674
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841132/HDFS-8674.trunk.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 9ceb14463035 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 3fd844b
          Default Java 1.8.0_111
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17715/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17715/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17715/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17715/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 15m 46s trunk passed +1 compile 2m 5s trunk passed +1 checkstyle 1m 23s trunk passed +1 mvnsite 1m 21s trunk passed +1 mvneclipse 0m 13s trunk passed -1 findbugs 1m 49s hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant Findbugs warnings. +1 javadoc 0m 41s trunk passed +1 mvninstall 0m 49s the patch passed +1 compile 0m 49s the patch passed +1 javac 0m 49s the patch passed +1 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 119 unchanged - 2 fixed = 119 total (was 121) +1 mvnsite 0m 54s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 56s the patch passed +1 javadoc 0m 39s the patch passed -1 unit 74m 34s hadoop-hdfs in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 105m 37s Reason Tests Failed junit tests hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks   hadoop.hdfs.server.datanode.TestDataNodeUUID   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness   hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation   hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker   hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes   hadoop.hdfs.server.datanode.TestDataNodeMXBean   hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR   hadoop.hdfs.TestLeaseRecoveryStriped   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.server.namenode.snapshot.TestSnapshot   hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication   hadoop.hdfs.server.namenode.ha.TestDNFencing Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-8674 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841132/HDFS-8674.trunk.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 9ceb14463035 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3fd844b Default Java 1.8.0_111 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17715/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/17715/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17715/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17715/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          kihwal Kihwal Lee added a comment -

          LightWeightLinkedSet does not support remove().

          Show
          kihwal Kihwal Lee added a comment - LightWeightLinkedSet does not support remove() .
          Hide
          daryn Daryn Sharp added a comment -

          Posted wrong patches. Changed the set from the lightweight back to standard one due to iterator issues.

          Show
          daryn Daryn Sharp added a comment - Posted wrong patches. Changed the set from the lightweight back to standard one due to iterator issues.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 6m 45s trunk passed
          +1 compile 0m 44s trunk passed
          +1 checkstyle 0m 26s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          -1 findbugs 1m 41s hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant Findbugs warnings.
          +1 javadoc 0m 39s trunk passed
          +1 mvninstall 0m 45s the patch passed
          +1 compile 0m 43s the patch passed
          +1 javac 0m 43s the patch passed
          +1 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 119 unchanged - 2 fixed = 119 total (was 121)
          +1 mvnsite 0m 48s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 45s the patch passed
          +1 javadoc 0m 36s the patch passed
          +1 unit 61m 9s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          79m 16s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-8674
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841300/HDFS-8674.trunk.2.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 06df29fbcd5a 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e0fa492
          Default Java 1.8.0_111
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17731/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17731/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17731/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 45s trunk passed +1 compile 0m 44s trunk passed +1 checkstyle 0m 26s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 12s trunk passed -1 findbugs 1m 41s hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant Findbugs warnings. +1 javadoc 0m 39s trunk passed +1 mvninstall 0m 45s the patch passed +1 compile 0m 43s the patch passed +1 javac 0m 43s the patch passed +1 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 119 unchanged - 2 fixed = 119 total (was 121) +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 45s the patch passed +1 javadoc 0m 36s the patch passed +1 unit 61m 9s hadoop-hdfs in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 79m 16s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-8674 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841300/HDFS-8674.trunk.2.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 06df29fbcd5a 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e0fa492 Default Java 1.8.0_111 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17731/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17731/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17731/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          kihwal Kihwal Lee added a comment -

          +1 The patch looks good.

          Show
          kihwal Kihwal Lee added a comment - +1 The patch looks good.
          Hide
          kihwal Kihwal Lee added a comment -

          Committed to trunk, branch-2 and branch-2.8. Thanks for working on this, Daryn.

          Show
          kihwal Kihwal Lee added a comment - Committed to trunk, branch-2 and branch-2.8. Thanks for working on this, Daryn.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10924 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10924/)
          HDFS-8674. Improve performance of postponed block scans. Contributed by (kihwal: rev 96c574927a600d15fab919df1fdc9e07887af6c5)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10924 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10924/ ) HDFS-8674 . Improve performance of postponed block scans. Contributed by (kihwal: rev 96c574927a600d15fab919df1fdc9e07887af6c5) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java

            People

            • Assignee:
              daryn Daryn Sharp
              Reporter:
              daryn Daryn Sharp
            • Votes:
              0 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development