Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8674

Improve performance of postponed block scans

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha2
    • Component/s: namenode
    • Labels:
      None

      Description

      When a standby goes active, it marks all nodes as "stale" which will cause block invalidations for over-replicated blocks to be queued until full block reports are received from the nodes with the block. The replication monitor scans the queue with O(N) runtime. It picks a random offset and iterates through the set to randomize blocks scanned.

      The result is devastating when a cluster loses multiple nodes during a rolling upgrade. Re-replication occurs, the nodes come back, the excess block invalidations are postponed. Rescanning just 2k blocks out of millions of postponed blocks may take multiple seconds. During the scan, the write lock is held which stalls all other processing.

      1. HDFS-8674.2.patch
        7 kB
        Daryn Sharp
      2. HDFS-8674.branch-2.patch
        7 kB
        Daryn Sharp
      3. HDFS-8674.patch
        6 kB
        Daryn Sharp
      4. HDFS-8674.patch
        6 kB
        Daryn Sharp
      5. HDFS-8674.trunk.2.patch
        8 kB
        Daryn Sharp
      6. HDFS-8674.trunk.patch
        8 kB
        Daryn Sharp

        Activity

        Hide
        daryn Daryn Sharp added a comment -

        No additional tests due to simple performance optimization that existing tests cover.

        Show
        daryn Daryn Sharp added a comment - No additional tests due to simple performance optimization that existing tests cover.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 15m 16s Findbugs (version ) appears to be broken on trunk.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 30s There were no new javac warning messages.
        +1 javadoc 9m 38s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 0m 50s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 35s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        -1 findbugs 3m 24s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
        +1 native 3m 14s Pre-build of native portion
        +1 hdfs tests 162m 31s Tests passed in hadoop-hdfs.
            204m 57s  



        Reason Tests
        FindBugs module:hadoop-hdfs



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12742164/HDFS-8674.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 60b858b
        Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/11502/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11502/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11502/testReport/
        Java 1.7.0_55
        uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11502/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 15m 16s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 30s There were no new javac warning messages. +1 javadoc 9m 38s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 50s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. -1 findbugs 3m 24s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 native 3m 14s Pre-build of native portion +1 hdfs tests 162m 31s Tests passed in hadoop-hdfs.     204m 57s   Reason Tests FindBugs module:hadoop-hdfs Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12742164/HDFS-8674.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 60b858b Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/11502/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11502/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11502/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11502/console This message was automatically generated.
        Hide
        mingma Ming Ma added a comment -

        Thanks Daryn Sharp!

        • The motivation of randomization is to make sure we don't end up scanning the same blocks if there isn't much update to postponedMisreplicatedBlocks. But maybe that doesn't matter much as it will eventually be drained; also HashSet doesn't make order guarantee.
        • Does the latency of couple seconds come from HashSet iteration? Some quick test indicates iterating through 5M entries of a HashSet take around 50ms.
        • What is the reason to change from HashSet to LinkedHashSet, to have some order guarantee? HashSet is faster than LinkedHashSet.
        • During the scan, if a block is marked as POSTPONE, it will be removed from postponedMisreplicatedBlocks first and add it back later via rescannedMisreplicatedBlocks. Can it just remove those blocks from postponedMisreplicatedBlocks that aren't marked as POSTPONE without using rescannedMisreplicatedBlocks?
        Show
        mingma Ming Ma added a comment - Thanks Daryn Sharp ! The motivation of randomization is to make sure we don't end up scanning the same blocks if there isn't much update to postponedMisreplicatedBlocks. But maybe that doesn't matter much as it will eventually be drained; also HashSet doesn't make order guarantee. Does the latency of couple seconds come from HashSet iteration? Some quick test indicates iterating through 5M entries of a HashSet take around 50ms. What is the reason to change from HashSet to LinkedHashSet, to have some order guarantee? HashSet is faster than LinkedHashSet. During the scan, if a block is marked as POSTPONE, it will be removed from postponedMisreplicatedBlocks first and add it back later via rescannedMisreplicatedBlocks. Can it just remove those blocks from postponedMisreplicatedBlocks that aren't marked as POSTPONE without using rescannedMisreplicatedBlocks?
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Daryn Sharp / Ming Ma, any update on this? Considering this for a 2.7.2 RC this weekend. Thanks.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Daryn Sharp / Ming Ma , any update on this? Considering this for a 2.7.2 RC this weekend. Thanks.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 patch 0m 0s The patch command could not apply the patch during dryrun.



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12742164/HDFS-8674.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 96677be
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13211/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12742164/HDFS-8674.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 96677be Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13211/console This message was automatically generated.
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Moving this out of 2.7.2 as there's been no update in a while.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Moving this out of 2.7.2 as there's been no update in a while.
        Hide
        daryn Daryn Sharp added a comment -

        Ming Ma It's a LinkedHashSet to cover the cases you state.

        The motivation of randomization is to make sure we don't end up scanning the same blocks if there isn't much update to postponedMisreplicatedBlocks. <...> What is the reason to change from HashSet to LinkedHashSet, to have some order guarantee? HashSet is faster than LinkedHashSet. <...> During the scan, if a block is marked as POSTPONE, it will be removed from postponedMisreplicatedBlocks first and add it back later via rescannedMisreplicatedBlocks. Can it just remove those blocks from postponedMisreplicatedBlocks that aren't marked as POSTPONE without using rescannedMisreplicatedBlocks?

        The LHS iterator is effectively functioning as a cheap circular array with acceptable insert/remove performance. Yes, the order guarantee ensures all blocks are visited w/o "skipping" to a random location. The block always has to be removed and re-inserted for the ordering to work. Otherwise the same blocks will be scanned over and over again.

        Does the latency of couple seconds come from HashSet iteration? Some quick test indicates iterating through 5M entries of a HashSet take around 50ms.

        Yes, the cycles wasted to skip the through the set were the killer. A micro-benchmark doesn't capture the same chaotic runtime environment of over a 100 threads competing for resources. Kihwal says (ironically) the performance "improved" under 5M.

        In multiple production incidents, the postponed queue backed up with 10-20M+ blocks. Scans determined the blocks are all still postponed. Pre-patch, the cycles took up to a few seconds and averaged in the many hundreds of ms. Performance was obviously atrocious. Post-patch, same scans took no more than a few ms.

        Show
        daryn Daryn Sharp added a comment - Ming Ma It's a LinkedHashSet to cover the cases you state. The motivation of randomization is to make sure we don't end up scanning the same blocks if there isn't much update to postponedMisreplicatedBlocks. <...> What is the reason to change from HashSet to LinkedHashSet, to have some order guarantee? HashSet is faster than LinkedHashSet. <...> During the scan, if a block is marked as POSTPONE, it will be removed from postponedMisreplicatedBlocks first and add it back later via rescannedMisreplicatedBlocks. Can it just remove those blocks from postponedMisreplicatedBlocks that aren't marked as POSTPONE without using rescannedMisreplicatedBlocks? The LHS iterator is effectively functioning as a cheap circular array with acceptable insert/remove performance. Yes, the order guarantee ensures all blocks are visited w/o "skipping" to a random location. The block always has to be removed and re-inserted for the ordering to work. Otherwise the same blocks will be scanned over and over again. Does the latency of couple seconds come from HashSet iteration? Some quick test indicates iterating through 5M entries of a HashSet take around 50ms. Yes, the cycles wasted to skip the through the set were the killer. A micro-benchmark doesn't capture the same chaotic runtime environment of over a 100 threads competing for resources. Kihwal says (ironically) the performance "improved" under 5M. In multiple production incidents, the postponed queue backed up with 10-20M+ blocks. Scans determined the blocks are all still postponed. Pre-patch, the cycles took up to a few seconds and averaged in the many hundreds of ms. Performance was obviously atrocious. Post-patch, same scans took no more than a few ms.
        Hide
        daryn Daryn Sharp added a comment -

        Ming Ma, comments?

        Show
        daryn Daryn Sharp added a comment - Ming Ma , comments?
        Hide
        mingma Ming Ma added a comment -

        Thanks Daryn Sharp. Your explanation makes sense. It is good to know the perf difference between "the # of items < 5M" and "larger number of items". +1 on the patch.

        Show
        mingma Ming Ma added a comment - Thanks Daryn Sharp . Your explanation makes sense. It is good to know the perf difference between "the # of items < 5M" and "larger number of items". +1 on the patch.
        Hide
        daryn Daryn Sharp added a comment -

        Updated patch since a data structure change prevented it from applying cleanly.

        Show
        daryn Daryn Sharp added a comment - Updated patch since a data structure change prevented it from applying cleanly.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 30s trunk passed
        +1 compile 0m 40s trunk passed with JDK v1.8.0_66
        +1 compile 0m 41s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 15s trunk passed
        +1 mvnsite 0m 51s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 1m 52s trunk passed
        +1 javadoc 1m 5s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 52s trunk passed with JDK v1.7.0_91
        +1 mvninstall 0m 48s the patch passed
        +1 compile 0m 38s the patch passed with JDK v1.8.0_66
        -1 javac 6m 16s hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66 with JDK v1.8.0_66 generated 1 new issues (was 32, now 32).
        +1 javac 0m 38s the patch passed
        +1 compile 0m 40s the patch passed with JDK v1.7.0_91
        -1 javac 6m 57s hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91 with JDK v1.7.0_91 generated 1 new issues (was 34, now 34).
        +1 javac 0m 40s the patch passed
        -1 checkstyle 0m 14s Patch generated 1 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 161, now 160).
        +1 mvnsite 0m 50s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 2s the patch passed
        +1 javadoc 1m 4s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 46s the patch passed with JDK v1.7.0_91
        -1 unit 52m 16s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        -1 unit 50m 8s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
        -1 asflicense 0m 25s Patch generated 58 ASF License warnings.
        128m 44s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestDFSClientRetries
          hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication
          hadoop.hdfs.server.namenode.TestDecommissioningStatus
          hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
          hadoop.hdfs.server.namenode.ha.TestDNFencing
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050
          hadoop.hdfs.TestDFSUpgradeFromImage
          hadoop.hdfs.TestSetrepDecreasing
          hadoop.hdfs.server.blockmanagement.TestNodeCount
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
          hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
        JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication
          hadoop.hdfs.TestLeaseRecovery2
          hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
          hadoop.hdfs.server.namenode.ha.TestDNFencing
          hadoop.hdfs.TestSetrepDecreasing
          hadoop.hdfs.server.blockmanagement.TestNodeCount
          hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778098/HDFS-8674.patch
        JIRA Issue HDFS-8674
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 4f4d2df9b67c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 0f708d4
        findbugs v3.0.0
        javac hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13903/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        javac hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HDFS-Build/13903/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13903/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13903/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13903/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13903/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13903/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13903/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13903/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Max memory used 75MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13903/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 30s trunk passed +1 compile 0m 40s trunk passed with JDK v1.8.0_66 +1 compile 0m 41s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 52s trunk passed +1 javadoc 1m 5s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 52s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 48s the patch passed +1 compile 0m 38s the patch passed with JDK v1.8.0_66 -1 javac 6m 16s hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66 with JDK v1.8.0_66 generated 1 new issues (was 32, now 32). +1 javac 0m 38s the patch passed +1 compile 0m 40s the patch passed with JDK v1.7.0_91 -1 javac 6m 57s hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91 with JDK v1.7.0_91 generated 1 new issues (was 34, now 34). +1 javac 0m 40s the patch passed -1 checkstyle 0m 14s Patch generated 1 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 161, now 160). +1 mvnsite 0m 50s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 2s the patch passed +1 javadoc 1m 4s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 46s the patch passed with JDK v1.7.0_91 -1 unit 52m 16s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 50m 8s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 25s Patch generated 58 ASF License warnings. 128m 44s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestDFSClientRetries   hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes   hadoop.hdfs.server.namenode.ha.TestDNFencing   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050   hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.TestSetrepDecreasing   hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication   hadoop.hdfs.TestLeaseRecovery2   hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes   hadoop.hdfs.server.namenode.ha.TestDNFencing   hadoop.hdfs.TestSetrepDecreasing   hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778098/HDFS-8674.patch JIRA Issue HDFS-8674 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 4f4d2df9b67c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0f708d4 findbugs v3.0.0 javac hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13903/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt javac hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HDFS-Build/13903/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13903/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13903/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13903/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13903/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13903/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13903/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13903/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 75MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13903/console This message was automatically generated.
        Hide
        mingma Ming Ma added a comment -

        Daryn, from failed tests's logs, the latest patch uses LightWeightLinkedSet whose iterator doesn't support remove.

        Show
        mingma Ming Ma added a comment - Daryn, from failed tests's logs, the latest patch uses LightWeightLinkedSet whose iterator doesn't support remove.
        Hide
        daryn Daryn Sharp added a comment -

        I'll try to rebase again.

        Show
        daryn Daryn Sharp added a comment - I'll try to rebase again.
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Daryn Sharp / Ming Ma, is it possible to get this in 2.7.3 in a week? Tx?

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Daryn Sharp / Ming Ma , is it possible to get this in 2.7.3 in a week? Tx?
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Haven't gotten any update, dropping this off into 2.7.4..

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Haven't gotten any update, dropping this off into 2.7.4..
        Hide
        andrew.wang Andrew Wang added a comment -

        Added some more target versions for tracking. Seems like this would still be good to get in.

        Show
        andrew.wang Andrew Wang added a comment - Added some more target versions for tracking. Seems like this would still be good to get in.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 patch 0m 6s HDFS-8674 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



        Subsystem Report/Notes
        JIRA Issue HDFS-8674
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778098/HDFS-8674.patch
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17194/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 6s HDFS-8674 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HDFS-8674 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778098/HDFS-8674.patch Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17194/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        daryn Daryn Sharp added a comment -

        I'll update this patch.

        Show
        daryn Daryn Sharp added a comment - I'll update this patch.
        Hide
        daryn Daryn Sharp added a comment -

        Resolved minor conflicts.

        Show
        daryn Daryn Sharp added a comment - Resolved minor conflicts.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 patch 0m 9s HDFS-8674 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



        Subsystem Report/Notes
        JIRA Issue HDFS-8674
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840925/HDFS-8674.2.patch
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17702/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 9s HDFS-8674 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HDFS-8674 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840925/HDFS-8674.2.patch Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17702/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        kihwal Kihwal Lee added a comment -

        Looks like HDFS-8674.2.patch is for branch-2. Can you post a trunk version?

        Show
        kihwal Kihwal Lee added a comment - Looks like HDFS-8674 .2.patch is for branch-2. Can you post a trunk version?
        Hide
        daryn Daryn Sharp added a comment -

        fix conflicts on trunk stemming from under replicated now being low redundancy.

        Show
        daryn Daryn Sharp added a comment - fix conflicts on trunk stemming from under replicated now being low redundancy.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 15m 46s trunk passed
        +1 compile 2m 5s trunk passed
        +1 checkstyle 1m 23s trunk passed
        +1 mvnsite 1m 21s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        -1 findbugs 1m 49s hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant Findbugs warnings.
        +1 javadoc 0m 41s trunk passed
        +1 mvninstall 0m 49s the patch passed
        +1 compile 0m 49s the patch passed
        +1 javac 0m 49s the patch passed
        +1 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 119 unchanged - 2 fixed = 119 total (was 121)
        +1 mvnsite 0m 54s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 56s the patch passed
        +1 javadoc 0m 39s the patch passed
        -1 unit 74m 34s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 21s The patch does not generate ASF License warnings.
        105m 37s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.blockmanagement.TestNodeCount
          hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
          hadoop.hdfs.server.datanode.TestDataNodeUUID
          hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
          hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness
          hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
          hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker
          hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
          hadoop.hdfs.server.datanode.TestDataNodeMXBean
          hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR
          hadoop.hdfs.TestLeaseRecoveryStriped
          hadoop.hdfs.server.datanode.TestDirectoryScanner
          hadoop.hdfs.server.namenode.snapshot.TestSnapshot
          hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication
          hadoop.hdfs.server.namenode.ha.TestDNFencing



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-8674
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841132/HDFS-8674.trunk.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 9ceb14463035 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 3fd844b
        Default Java 1.8.0_111
        findbugs v3.0.0
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17715/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17715/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17715/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17715/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 15m 46s trunk passed +1 compile 2m 5s trunk passed +1 checkstyle 1m 23s trunk passed +1 mvnsite 1m 21s trunk passed +1 mvneclipse 0m 13s trunk passed -1 findbugs 1m 49s hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant Findbugs warnings. +1 javadoc 0m 41s trunk passed +1 mvninstall 0m 49s the patch passed +1 compile 0m 49s the patch passed +1 javac 0m 49s the patch passed +1 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 119 unchanged - 2 fixed = 119 total (was 121) +1 mvnsite 0m 54s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 56s the patch passed +1 javadoc 0m 39s the patch passed -1 unit 74m 34s hadoop-hdfs in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 105m 37s Reason Tests Failed junit tests hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks   hadoop.hdfs.server.datanode.TestDataNodeUUID   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness   hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation   hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker   hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes   hadoop.hdfs.server.datanode.TestDataNodeMXBean   hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR   hadoop.hdfs.TestLeaseRecoveryStriped   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.server.namenode.snapshot.TestSnapshot   hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication   hadoop.hdfs.server.namenode.ha.TestDNFencing Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-8674 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841132/HDFS-8674.trunk.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 9ceb14463035 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3fd844b Default Java 1.8.0_111 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17715/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/17715/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17715/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17715/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        kihwal Kihwal Lee added a comment -

        LightWeightLinkedSet does not support remove().

        Show
        kihwal Kihwal Lee added a comment - LightWeightLinkedSet does not support remove() .
        Hide
        daryn Daryn Sharp added a comment -

        Posted wrong patches. Changed the set from the lightweight back to standard one due to iterator issues.

        Show
        daryn Daryn Sharp added a comment - Posted wrong patches. Changed the set from the lightweight back to standard one due to iterator issues.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 10s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 45s trunk passed
        +1 compile 0m 44s trunk passed
        +1 checkstyle 0m 26s trunk passed
        +1 mvnsite 0m 51s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        -1 findbugs 1m 41s hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant Findbugs warnings.
        +1 javadoc 0m 39s trunk passed
        +1 mvninstall 0m 45s the patch passed
        +1 compile 0m 43s the patch passed
        +1 javac 0m 43s the patch passed
        +1 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 119 unchanged - 2 fixed = 119 total (was 121)
        +1 mvnsite 0m 48s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 45s the patch passed
        +1 javadoc 0m 36s the patch passed
        +1 unit 61m 9s hadoop-hdfs in the patch passed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        79m 16s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-8674
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841300/HDFS-8674.trunk.2.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 06df29fbcd5a 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / e0fa492
        Default Java 1.8.0_111
        findbugs v3.0.0
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17731/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17731/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17731/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 45s trunk passed +1 compile 0m 44s trunk passed +1 checkstyle 0m 26s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 12s trunk passed -1 findbugs 1m 41s hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant Findbugs warnings. +1 javadoc 0m 39s trunk passed +1 mvninstall 0m 45s the patch passed +1 compile 0m 43s the patch passed +1 javac 0m 43s the patch passed +1 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 119 unchanged - 2 fixed = 119 total (was 121) +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 45s the patch passed +1 javadoc 0m 36s the patch passed +1 unit 61m 9s hadoop-hdfs in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 79m 16s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-8674 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841300/HDFS-8674.trunk.2.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 06df29fbcd5a 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e0fa492 Default Java 1.8.0_111 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17731/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17731/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17731/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        kihwal Kihwal Lee added a comment -

        +1 The patch looks good.

        Show
        kihwal Kihwal Lee added a comment - +1 The patch looks good.
        Hide
        kihwal Kihwal Lee added a comment -

        Committed to trunk, branch-2 and branch-2.8. Thanks for working on this, Daryn.

        Show
        kihwal Kihwal Lee added a comment - Committed to trunk, branch-2 and branch-2.8. Thanks for working on this, Daryn.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10924 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10924/)
        HDFS-8674. Improve performance of postponed block scans. Contributed by (kihwal: rev 96c574927a600d15fab919df1fdc9e07887af6c5)

        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10924 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10924/ ) HDFS-8674 . Improve performance of postponed block scans. Contributed by (kihwal: rev 96c574927a600d15fab919df1fdc9e07887af6c5) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java

          People

          • Assignee:
            daryn Daryn Sharp
            Reporter:
            daryn Daryn Sharp
          • Votes:
            0 Vote for this issue
            Watchers:
            18 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development