Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11634

Optimize BlockIterator when iterating starts in the middle.

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.5
    • Fix Version/s: 2.9.0, 2.7.4, 3.0.0-alpha4, 2.8.2
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      BlockManager.getBlocksWithLocations() needs to iterate blocks from a randomly selected startBlock index. It creates an iterator which points to the first block and then skips all blocks until startBlock. It is inefficient when DN has multiple storages. Instead of skipping blocks one by one we can skip entire storages. Should be more efficient on average.

      1. Screen Shot 2017-06-21 at 1.39.21 PM.png
        124 kB
        Zhe Zhang
      2. HDFS-11643.005.patch
        11 kB
        Konstantin Shvachko
      3. HDFS-11634-branch-2.7.006.patch
        12 kB
        Konstantin Shvachko
      4. HDFS-11634.006.patch
        11 kB
        Konstantin Shvachko
      5. HDFS-11634.004.patch
        12 kB
        Konstantin Shvachko
      6. HDFS-11634.003.patch
        12 kB
        Konstantin Shvachko
      7. HDFS-11634.002.patch
        12 kB
        Konstantin Shvachko
      8. HDFS-11634.001.patch
        3 kB
        Konstantin Shvachko

        Issue Links

          Activity

          Hide
          shv Konstantin Shvachko added a comment -

          Simple optimization.

          Show
          shv Konstantin Shvachko added a comment - Simple optimization.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 23s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 14m 9s trunk passed
          +1 compile 0m 55s trunk passed
          +1 checkstyle 0m 41s trunk passed
          +1 mvnsite 0m 59s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 2m 2s trunk passed
          +1 javadoc 0m 45s trunk passed
          +1 mvninstall 0m 55s the patch passed
          +1 compile 0m 51s the patch passed
          +1 javac 0m 51s the patch passed
          -0 checkstyle 0m 39s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 147 unchanged - 0 fixed = 148 total (was 147)
          +1 mvnsite 0m 58s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 1s The patch has no whitespace issues.
          +1 findbugs 2m 10s the patch passed
          +1 javadoc 0m 46s the patch passed
          -1 unit 73m 6s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          101m 42s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11634
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12862554/HDFS-11634.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 8f18acb67771 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2aa8967
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19021/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/19021/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19021/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19021/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 14m 9s trunk passed +1 compile 0m 55s trunk passed +1 checkstyle 0m 41s trunk passed +1 mvnsite 0m 59s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 2m 2s trunk passed +1 javadoc 0m 45s trunk passed +1 mvninstall 0m 55s the patch passed +1 compile 0m 51s the patch passed +1 javac 0m 51s the patch passed -0 checkstyle 0m 39s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 147 unchanged - 0 fixed = 148 total (was 147) +1 mvnsite 0m 58s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 1s The patch has no whitespace issues. +1 findbugs 2m 10s the patch passed +1 javadoc 0m 46s the patch passed -1 unit 73m 6s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 101m 42s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11634 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12862554/HDFS-11634.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8f18acb67771 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2aa8967 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19021/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19021/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19021/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19021/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          shahrs87 Rushabh S Shah added a comment -

          Overall the patch looks good.
          But I have one question.
          In the patch , should the line if(numBlocks < s) { be if(numBlocks '<=' s) { ?

            if(numBlocks < s) {
                   index++;
                   s -= numBlocks;
                 }
          
          Show
          shahrs87 Rushabh S Shah added a comment - Overall the patch looks good. But I have one question. In the patch , should the line if(numBlocks < s) { be if(numBlocks '<=' s) { ? if(numBlocks < s) { index++; s -= numBlocks; }
          Hide
          shahrs87 Rushabh S Shah added a comment -

          Just FYI,
          TestFsDatasetImpl.testLoadingDfsUsedForVolumes test failure is being tracked by HADOOP-14293

          Show
          shahrs87 Rushabh S Shah added a comment - Just FYI, TestFsDatasetImpl.testLoadingDfsUsedForVolumes test failure is being tracked by HADOOP-14293
          Hide
          shv Konstantin Shvachko added a comment -

          Good catch Rushabh S Shah. I corrected corner cases and added a unit test for BlockIterator, which verifies them now.

          Show
          shv Konstantin Shvachko added a comment - Good catch Rushabh S Shah . I corrected corner cases and added a unit test for BlockIterator, which verifies them now.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 15m 9s trunk passed
          +1 compile 0m 55s trunk passed
          +1 checkstyle 0m 37s trunk passed
          +1 mvnsite 0m 55s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 1m 46s trunk passed
          +1 javadoc 0m 41s trunk passed
          +1 mvninstall 0m 47s the patch passed
          +1 compile 0m 44s the patch passed
          +1 javac 0m 44s the patch passed
          -0 checkstyle 0m 35s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 183 unchanged - 0 fixed = 185 total (was 183)
          +1 mvnsite 0m 49s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 52s the patch passed
          +1 javadoc 0m 37s the patch passed
          -1 unit 69m 30s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          97m 20s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:612578f
          JIRA Issue HDFS-11634
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12862761/HDFS-11634.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux a658625c9e36 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 7999318a
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19035/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/19035/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19035/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19035/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 15m 9s trunk passed +1 compile 0m 55s trunk passed +1 checkstyle 0m 37s trunk passed +1 mvnsite 0m 55s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 46s trunk passed +1 javadoc 0m 41s trunk passed +1 mvninstall 0m 47s the patch passed +1 compile 0m 44s the patch passed +1 javac 0m 44s the patch passed -0 checkstyle 0m 35s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 183 unchanged - 0 fixed = 185 total (was 183) +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 52s the patch passed +1 javadoc 0m 37s the patch passed -1 unit 69m 30s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 97m 20s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting Subsystem Report/Notes Docker Image:yetus/hadoop:612578f JIRA Issue HDFS-11634 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12862761/HDFS-11634.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a658625c9e36 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7999318a Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19035/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19035/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19035/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19035/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          shahrs87 Rushabh S Shah added a comment - - edited

          One Minor nit:
          Couple of typos for 'oterator'

          One question:
          I don't understand why sumBlocks was introduced in DatanodeDescriptor#BlockIterator constructor in latest patch.
          What corner case were you trying to fix ?

          Show
          shahrs87 Rushabh S Shah added a comment - - edited One Minor nit: Couple of typos for 'oterator' One question: I don't understand why sumBlocks was introduced in DatanodeDescriptor#BlockIterator constructor in latest patch. What corner case were you trying to fix ?
          Hide
          shv Konstantin Shvachko added a comment -
          • Fixed the typos. Thanks.
          • Yes this the corner case.
            In the previous patch if I have 3 storages having {3, 3, 1} blocks respectively. And I want to set iterator to startBlock=2. Then s=2 <= numBlocks for the first two storages, but not the third, and index will increment. Which is incorrect as startBlock=2 is on storage #0, rather than #1. My solution is to base the if condition directly on startBlock, and then one should accumulate blocks in storages, which sumBlocks does.
            Hope this makes sense.
          Show
          shv Konstantin Shvachko added a comment - Fixed the typos. Thanks. Yes this the corner case. In the previous patch if I have 3 storages having {3, 3, 1} blocks respectively. And I want to set iterator to startBlock=2. Then s=2 <= numBlocks for the first two storages, but not the third, and index will increment. Which is incorrect as startBlock=2 is on storage #0, rather than #1. My solution is to base the if condition directly on startBlock, and then one should accumulate blocks in storages, which sumBlocks does. Hope this makes sense.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 13m 24s trunk passed
          +1 compile 0m 59s trunk passed
          +1 checkstyle 0m 37s trunk passed
          +1 mvnsite 0m 58s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 1m 59s trunk passed
          +1 javadoc 0m 43s trunk passed
          +1 mvninstall 0m 59s the patch passed
          +1 compile 0m 56s the patch passed
          +1 javac 0m 56s the patch passed
          -0 checkstyle 0m 38s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 183 unchanged - 0 fixed = 185 total (was 183)
          +1 mvnsite 1m 2s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 4s the patch passed
          +1 javadoc 0m 36s the patch passed
          -1 unit 68m 34s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          96m 4s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.server.namenode.TestNamenodeCapacityReport



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:612578f
          JIRA Issue HDFS-11634
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12862961/HDFS-11634.003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 74ff1e0c3974 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 3a91376
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19057/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/19057/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19057/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19057/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 13m 24s trunk passed +1 compile 0m 59s trunk passed +1 checkstyle 0m 37s trunk passed +1 mvnsite 0m 58s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 59s trunk passed +1 javadoc 0m 43s trunk passed +1 mvninstall 0m 59s the patch passed +1 compile 0m 56s the patch passed +1 javac 0m 56s the patch passed -0 checkstyle 0m 38s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 183 unchanged - 0 fixed = 185 total (was 183) +1 mvnsite 1m 2s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 4s the patch passed +1 javadoc 0m 36s the patch passed -1 unit 68m 34s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 96m 4s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport Subsystem Report/Notes Docker Image:yetus/hadoop:612578f JIRA Issue HDFS-11634 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12862961/HDFS-11634.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 74ff1e0c3974 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3a91376 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19057/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19057/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19057/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19057/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          shahrs87 Rushabh S Shah added a comment -

          Hope this makes sense.

          Makes perfect sense.
          +1 (non-binding).
          There are couple of checkstyle warnings carried over from patch#2.
          I wouldn't worry about submitting a new patch and wasting build resources. The committer can fix it while committing.
          Konstantin Shvachko: thanks for the fix.
          Hopefully some committer can review and commit.

          Show
          shahrs87 Rushabh S Shah added a comment - Hope this makes sense. Makes perfect sense. +1 (non-binding). There are couple of checkstyle warnings carried over from patch#2. I wouldn't worry about submitting a new patch and wasting build resources. The committer can fix it while committing. Konstantin Shvachko : thanks for the fix. Hopefully some committer can review and commit.
          Hide
          shv Konstantin Shvachko added a comment -

          Addressing checkstyle warnings.
          Thanks for the review Rushabh S Shah.

          Show
          shv Konstantin Shvachko added a comment - Addressing checkstyle warnings. Thanks for the review Rushabh S Shah .
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 17m 39s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 15m 8s trunk passed
          +1 compile 0m 55s trunk passed
          +1 checkstyle 0m 41s trunk passed
          +1 mvnsite 0m 59s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 findbugs 1m 58s trunk passed
          +1 javadoc 0m 43s trunk passed
          +1 mvninstall 0m 56s the patch passed
          +1 compile 0m 51s the patch passed
          +1 javac 0m 51s the patch passed
          +1 checkstyle 0m 37s the patch passed
          +1 mvnsite 0m 57s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 4s the patch passed
          +1 javadoc 0m 41s the patch passed
          +1 unit 70m 31s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          116m 58s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:612578f
          JIRA Issue HDFS-11634
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12863143/HDFS-11634.004.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux f7bf345c46a3 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / b053fdc
          Default Java 1.8.0_121
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19066/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19066/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 17m 39s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 15m 8s trunk passed +1 compile 0m 55s trunk passed +1 checkstyle 0m 41s trunk passed +1 mvnsite 0m 59s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 1m 58s trunk passed +1 javadoc 0m 43s trunk passed +1 mvninstall 0m 56s the patch passed +1 compile 0m 51s the patch passed +1 javac 0m 51s the patch passed +1 checkstyle 0m 37s the patch passed +1 mvnsite 0m 57s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 4s the patch passed +1 javadoc 0m 41s the patch passed +1 unit 70m 31s hadoop-hdfs in the patch passed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 116m 58s Subsystem Report/Notes Docker Image:yetus/hadoop:612578f JIRA Issue HDFS-11634 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12863143/HDFS-11634.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f7bf345c46a3 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b053fdc Default Java 1.8.0_121 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19066/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19066/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Konstantin Shvachko for the patch and Rushabh S Shah for the review. The idea looks good to me, it's a good improvement to skip entire storages.

          I only have one concern/question:

                for (DatanodeStorageInfo e : storages) {
                  iterators.add(e.getBlockIterator());
                  int numBlocks = e.numBlocks();
                  sumBlocks += numBlocks;
                  if(sumBlocks <= startBlock) {
                    index++;
                    s -= numBlocks;
                  }
                }
          

          If a storage is skipped, should we still add it to the iterators?

          Show
          zhz Zhe Zhang added a comment - Thanks Konstantin Shvachko for the patch and Rushabh S Shah for the review. The idea looks good to me, it's a good improvement to skip entire storages. I only have one concern/question: for (DatanodeStorageInfo e : storages) { iterators.add(e.getBlockIterator()); int numBlocks = e.numBlocks(); sumBlocks += numBlocks; if (sumBlocks <= startBlock) { index++; s -= numBlocks; } } If a storage is skipped, should we still add it to the iterators ?
          Hide
          redvine Vinitha Reddy Gankidi added a comment -

          It's a good improvement. One minor nit:
          index is initialized to zero twice

          Zhe Zhang raised a good point. It seems like we don't need the iterators for the skipped storages.

          Show
          redvine Vinitha Reddy Gankidi added a comment - It's a good improvement. One minor nit: index is initialized to zero twice Zhe Zhang raised a good point. It seems like we don't need the iterators for the skipped storages.
          Hide
          shv Konstantin Shvachko added a comment -

          Agreed, since we never iterate backwards, we don't need iterators from skipped storges.
          Latest patch addresses both of your comments guys. Plus I simplified the test and made it verify all possible combinations.

          Show
          shv Konstantin Shvachko added a comment - Agreed, since we never iterate backwards, we don't need iterators from skipped storges. Latest patch addresses both of your comments guys. Plus I simplified the test and made it verify all possible combinations.
          Hide
          shahrs87 Rushabh S Shah added a comment -

          Agreed, since we never iterate backwards, we don't need iterators from skipped storges.

          We do iterate backwards when the requested data size is more than the fetched size (from the offset we chose randomly).
          It creates a brand new iterator there and it would be nice if we can use the same iterator which we created above by resetting some index.

            if(totalSize<size) {
                iter = node.getBlockIterator(); // start from the beginning
                for(int i=0; i<startBlock&&totalSize<size; i++) {
                  curBlock = iter.next();
                  if(!curBlock.isComplete())  continue;
                  if (curBlock.getNumBytes() < getBlocksMinBlockSize) {
                    continue;
                  }
                  totalSize += addBlock(curBlock, results);
                }
          
          Show
          shahrs87 Rushabh S Shah added a comment - Agreed, since we never iterate backwards, we don't need iterators from skipped storges. We do iterate backwards when the requested data size is more than the fetched size (from the offset we chose randomly). It creates a brand new iterator there and it would be nice if we can use the same iterator which we created above by resetting some index. if(totalSize<size) { iter = node.getBlockIterator(); // start from the beginning for(int i=0; i<startBlock&&totalSize<size; i++) { curBlock = iter.next(); if(!curBlock.isComplete()) continue; if (curBlock.getNumBytes() < getBlocksMinBlockSize) { continue; } totalSize += addBlock(curBlock, results); }
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 16m 22s trunk passed
          +1 compile 0m 55s trunk passed
          +1 checkstyle 0m 41s trunk passed
          +1 mvnsite 1m 11s trunk passed
          +1 mvneclipse 0m 19s trunk passed
          +1 findbugs 2m 7s trunk passed
          +1 javadoc 0m 47s trunk passed
          +1 mvninstall 0m 59s the patch passed
          +1 compile 0m 57s the patch passed
          +1 javac 0m 57s the patch passed
          +1 checkstyle 0m 40s the patch passed
          +1 mvnsite 1m 8s the patch passed
          +1 mvneclipse 0m 15s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 19s the patch passed
          +1 javadoc 0m 45s the patch passed
          -1 unit 83m 7s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 25s The patch does not generate ASF License warnings.
          114m 50s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.TestEncryptionZonesWithKMS



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:612578f
          JIRA Issue HDFS-11634
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12863379/HDFS-11643.005.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 35e9236b0f3f 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0cab572
          Default Java 1.8.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/19086/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19086/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19086/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 16m 22s trunk passed +1 compile 0m 55s trunk passed +1 checkstyle 0m 41s trunk passed +1 mvnsite 1m 11s trunk passed +1 mvneclipse 0m 19s trunk passed +1 findbugs 2m 7s trunk passed +1 javadoc 0m 47s trunk passed +1 mvninstall 0m 59s the patch passed +1 compile 0m 57s the patch passed +1 javac 0m 57s the patch passed +1 checkstyle 0m 40s the patch passed +1 mvnsite 1m 8s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 19s the patch passed +1 javadoc 0m 45s the patch passed -1 unit 83m 7s hadoop-hdfs in the patch failed. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 114m 50s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.TestEncryptionZonesWithKMS Subsystem Report/Notes Docker Image:yetus/hadoop:612578f JIRA Issue HDFS-11634 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12863379/HDFS-11643.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 35e9236b0f3f 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0cab572 Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/19086/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19086/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19086/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          shv Konstantin Shvachko added a comment -

          Hey Rushabh Shah, what I mean is that Iterator interface does not have API to iterate backward or reset the iterator to a random member of a collections.
          The snippet you are referring to indeed loops back to the beginning, but that should be pretty rare, when the initial random starting point was chosen too close to the end of the list. I don't think we need to optimize the circular jump. Do you?

          Show
          shv Konstantin Shvachko added a comment - Hey Rushabh Shah , what I mean is that Iterator interface does not have API to iterate backward or reset the iterator to a random member of a collections. The snippet you are referring to indeed loops back to the beginning, but that should be pretty rare, when the initial random starting point was chosen too close to the end of the list. I don't think we need to optimize the circular jump. Do you?
          Hide
          shahrs87 Rushabh S Shah added a comment -

          I don't think we need to optimize the circular jump. Do you?

          No. It was just a thought.
          It is no more worse than what is today.
          Just to clarify, I don't have any objections to the latest version of the patch.
          +1 non-binding.

          Show
          shahrs87 Rushabh S Shah added a comment - I don't think we need to optimize the circular jump. Do you? No. It was just a thought. It is no more worse than what is today. Just to clarify, I don't have any objections to the latest version of the patch. +1 non-binding.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Konstantin for updating the patch.

          v5 patch LGTM. +1. Nice optimization!

          Show
          zhz Zhe Zhang added a comment - Thanks Konstantin for updating the patch. v5 patch LGTM. +1. Nice optimization!
          Hide
          shv Konstantin Shvachko added a comment -

          Minor correction of the new test to ease porting to branch-2.7.
          And submitting the patch for branch-2.7.

          Show
          shv Konstantin Shvachko added a comment - Minor correction of the new test to ease porting to branch-2.7. And submitting the patch for branch-2.7.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11597 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11597/)
          HDFS-11634. Optimize BlockIterator when interating starts in the middle. (shv: rev 8dfcd95d580bb090af7f40af0a57061518c18c8c)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestGetBlocks.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11597 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11597/ ) HDFS-11634 . Optimize BlockIterator when interating starts in the middle. (shv: rev 8dfcd95d580bb090af7f40af0a57061518c18c8c) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestGetBlocks.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 12m 22s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 8m 53s branch-2.7 passed
          +1 compile 1m 17s branch-2.7 passed with JDK v1.8.0_121
          +1 compile 1m 10s branch-2.7 passed with JDK v1.7.0_121
          +1 checkstyle 0m 29s branch-2.7 passed
          +1 mvnsite 1m 9s branch-2.7 passed
          +1 mvneclipse 0m 17s branch-2.7 passed
          -1 findbugs 3m 13s hadoop-hdfs-project/hadoop-hdfs in branch-2.7 has 1 extant Findbugs warnings.
          +1 javadoc 1m 4s branch-2.7 passed with JDK v1.8.0_121
          +1 javadoc 1m 51s branch-2.7 passed with JDK v1.7.0_121
          +1 mvninstall 0m 57s the patch passed
          +1 compile 1m 3s the patch passed with JDK v1.8.0_121
          +1 javac 1m 3s the patch passed
          +1 compile 1m 3s the patch passed with JDK v1.7.0_121
          +1 javac 1m 3s the patch passed
          -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 238 unchanged - 1 fixed = 239 total (was 239)
          +1 mvnsite 0m 58s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          -1 whitespace 0m 0s The patch has 2688 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          -1 whitespace 1m 8s The patch 74 line(s) with tabs.
          +1 findbugs 3m 16s the patch passed
          +1 javadoc 0m 59s the patch passed with JDK v1.8.0_121
          +1 javadoc 1m 47s the patch passed with JDK v1.7.0_121
          -1 unit 45m 59s hadoop-hdfs in the patch failed with JDK v1.7.0_121.
          -1 asflicense 0m 19s The patch generated 3 ASF License warnings.
          147m 6s



          Reason Tests
          JDK v1.8.0_121 Failed junit tests hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
            hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.util.TestByteArrayManager
          JDK v1.7.0_121 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:c420dfe
          JIRA Issue HDFS-11634
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12863708/HDFS-11634-branch-2.7.006.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 3ed6cc5ca354 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2.7 / ccd24f0
          Default Java 1.7.0_121
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19117/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19117/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19117/artifact/patchprocess/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19117/artifact/patchprocess/whitespace-tabs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/19117/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_121.txt
          JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19117/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/19117/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19117/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 12m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 8m 53s branch-2.7 passed +1 compile 1m 17s branch-2.7 passed with JDK v1.8.0_121 +1 compile 1m 10s branch-2.7 passed with JDK v1.7.0_121 +1 checkstyle 0m 29s branch-2.7 passed +1 mvnsite 1m 9s branch-2.7 passed +1 mvneclipse 0m 17s branch-2.7 passed -1 findbugs 3m 13s hadoop-hdfs-project/hadoop-hdfs in branch-2.7 has 1 extant Findbugs warnings. +1 javadoc 1m 4s branch-2.7 passed with JDK v1.8.0_121 +1 javadoc 1m 51s branch-2.7 passed with JDK v1.7.0_121 +1 mvninstall 0m 57s the patch passed +1 compile 1m 3s the patch passed with JDK v1.8.0_121 +1 javac 1m 3s the patch passed +1 compile 1m 3s the patch passed with JDK v1.7.0_121 +1 javac 1m 3s the patch passed -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 238 unchanged - 1 fixed = 239 total (was 239) +1 mvnsite 0m 58s the patch passed +1 mvneclipse 0m 13s the patch passed -1 whitespace 0m 0s The patch has 2688 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 whitespace 1m 8s The patch 74 line(s) with tabs. +1 findbugs 3m 16s the patch passed +1 javadoc 0m 59s the patch passed with JDK v1.8.0_121 +1 javadoc 1m 47s the patch passed with JDK v1.7.0_121 -1 unit 45m 59s hadoop-hdfs in the patch failed with JDK v1.7.0_121. -1 asflicense 0m 19s The patch generated 3 ASF License warnings. 147m 6s Reason Tests JDK v1.8.0_121 Failed junit tests hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.util.TestByteArrayManager JDK v1.7.0_121 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots Subsystem Report/Notes Docker Image:yetus/hadoop:c420dfe JIRA Issue HDFS-11634 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12863708/HDFS-11634-branch-2.7.006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 3ed6cc5ca354 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / ccd24f0 Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19117/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19117/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19117/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19117/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19117/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_121.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19117/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/19117/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19117/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          shv Konstantin Shvachko added a comment -

          All tests Jenkins failed for 2.7 passed locally.

          Show
          shv Konstantin Shvachko added a comment - All tests Jenkins failed for 2.7 passed locally.
          Hide
          shv Konstantin Shvachko added a comment -

          Just committed this to trunk, branch-2, 2.8, and 2.7.
          Thanks for constructive reviews, everybody.

          Show
          shv Konstantin Shvachko added a comment - Just committed this to trunk, branch-2, 2.8, and 2.7. Thanks for constructive reviews, everybody.
          Hide
          zhz Zhe Zhang added a comment - - edited

          On one of our clusters with ~150 million blocks, getBlocks used to take 40ms. With the change it is taking 9ms, more than 4x faster! This also leads to a significant improvement on NameNode RPC performance when Balancer is running.

          Show
          zhz Zhe Zhang added a comment - - edited On one of our clusters with ~150 million blocks, getBlocks used to take 40ms. With the change it is taking 9ms, more than 4x faster! This also leads to a significant improvement on NameNode RPC performance when Balancer is running.
          Hide
          shv Konstantin Shvachko added a comment -

          Good job measuring this, Zhe Zhang. Thanks.
          The main outcome here I think is substantial reduction in NN lock holding time for internal (housekeeping) load.

          Show
          shv Konstantin Shvachko added a comment - Good job measuring this, Zhe Zhang . Thanks. The main outcome here I think is substantial reduction in NN lock holding time for internal (housekeeping) load.

            People

            • Assignee:
              shv Konstantin Shvachko
              Reporter:
              shv Konstantin Shvachko
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development