Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      1) Hung node detection: HDFS-6247 has removed the socket read timeout while adding the periodic response for slow block moves. However, the removal of the long timeout wasn't necessary. The timeout is still useful for avoiding hung nodes and does not abort slow moves.

      2) Enforcing the iteration limit:The 20 minute iteration limit is supposed to be enforced, but it is not. An iteration can easily stretch to 30 to 40 minutes with a long tail. Because of the long tails, the balancer throughput does not reach its full potential.

      3) Slow move detection: For improved throughput, imposing block move timeout is sometimes necessary. We have seen an iteration taking over 2 hours because of one slow block move. This is mainly for catching exceptionally slow moves. Even if the balancer stops waiting, the move will continue and finish.

      In order to not undo what HDFS-6247 tried to achieve, it should be possible to configure off 3).

      1. balancer.png
        94 kB
        Kihwal Lee
      2. HDFS-11015-1.patch
        7 kB
        Kihwal Lee
      3. HDFS-11015-2.patch
        9 kB
        Kihwal Lee
      4. HDFS-11015-3.patch
        9 kB
        Kihwal Lee

        Issue Links

          Activity

          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 8m 1s trunk passed
          +1 compile 0m 47s trunk passed
          +1 checkstyle 0m 32s trunk passed
          +1 mvnsite 0m 57s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 48s trunk passed
          +1 javadoc 0m 40s trunk passed
          +1 mvninstall 0m 49s the patch passed
          +1 compile 0m 46s the patch passed
          +1 javac 0m 46s the patch passed
          -0 checkstyle 0m 27s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 440 unchanged - 0 fixed = 443 total (was 440)
          +1 mvnsite 0m 51s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 1m 55s the patch passed
          +1 javadoc 0m 38s the patch passed
          +1 unit 59m 2s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          79m 28s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-11015
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833775/HDFS-11015-1.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 82c0b333ffc0 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ed9fcbe
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17184/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/17184/artifact/patchprocess/whitespace-eol.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17184/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17184/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 1s trunk passed +1 compile 0m 47s trunk passed +1 checkstyle 0m 32s trunk passed +1 mvnsite 0m 57s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 48s trunk passed +1 javadoc 0m 40s trunk passed +1 mvninstall 0m 49s the patch passed +1 compile 0m 46s the patch passed +1 javac 0m 46s the patch passed -0 checkstyle 0m 27s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 440 unchanged - 0 fixed = 443 total (was 440) +1 mvnsite 0m 51s the patch passed +1 mvneclipse 0m 11s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 1m 55s the patch passed +1 javadoc 0m 38s the patch passed +1 unit 59m 2s hadoop-hdfs in the patch passed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 79m 28s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-11015 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833775/HDFS-11015-1.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 82c0b333ffc0 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ed9fcbe Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17184/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/17184/artifact/patchprocess/whitespace-eol.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17184/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17184/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks for the patch Kihwal. A while ago I reported a similar issue HDFS-10977.

          Show
          zhz Zhe Zhang added a comment - Thanks for the patch Kihwal. A while ago I reported a similar issue HDFS-10977 .
          Hide
          zhz Zhe Zhang added a comment -

          Mark Wagner Has been working on a related issue HDFS-10966.

          Show
          zhz Zhe Zhang added a comment - Mark Wagner Has been working on a related issue HDFS-10966 .
          Hide
          kihwal Kihwal Lee added a comment -

          Here is the graph that shows the improvement from an expanding (i.e. a lot to balance) big cluster. The thing long tails are eliminated by adding block move timeout and read timeout. The multi-minute ramp-down tail is gone after enforcing the iteration limit.

          Show
          kihwal Kihwal Lee added a comment - Here is the graph that shows the improvement from an expanding (i.e. a lot to balance) big cluster. The thing long tails are eliminated by adding block move timeout and read timeout. The multi-minute ramp-down tail is gone after enforcing the iteration limit.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Kihwal for sharing the results; they look really interesting!

          However, the removal of the long timeout wasn't necessary. The timeout is still useful for avoiding hung nodes and does not abort slow moves.

          I agree. We should use a combination of IN_PROGRESS heartbeating and socket timeout.

          Regarding DFS_BALANCER_BLOCK_MOVE_TIMEOUT:

          1. Since it's named and documented as a Balancer config knob, I guess we should initiate it inside Balancer and pass it to Dispatcher? Otherwise Mover is silently affected by this config.
          2. Nit: "3 to 5 minute" should be "3 to 5 minutes"

          Regarding the 20 mins timeout:

          1. There are only two while loops in Dispatcher so I think your patch does a pretty good job in enforcing the timeout. Can we simplify the loops by merging the conditions into the while statement? Especially the one in dispatchBlocks, isTimeUp isn't necessary.

          Overall the patch looks pretty good. I realized that it doesn't really cover the HDFS-10977 scenario. May I suggest we update the subject to "Enforce iteration time limit and slow block move timeout in Balancer"? It doesn't look easy to add a unit test and I'm OK in this case because the added logic is pretty straightforward. I'm happy to open a separate test JIRA though.

          Show
          zhz Zhe Zhang added a comment - Thanks Kihwal for sharing the results; they look really interesting! However, the removal of the long timeout wasn't necessary. The timeout is still useful for avoiding hung nodes and does not abort slow moves. I agree. We should use a combination of IN_PROGRESS heartbeating and socket timeout. Regarding DFS_BALANCER_BLOCK_MOVE_TIMEOUT : Since it's named and documented as a Balancer config knob, I guess we should initiate it inside Balancer and pass it to Dispatcher ? Otherwise Mover is silently affected by this config. Nit: "3 to 5 minute" should be "3 to 5 minutes" Regarding the 20 mins timeout: There are only two while loops in Dispatcher so I think your patch does a pretty good job in enforcing the timeout. Can we simplify the loops by merging the conditions into the while statement? Especially the one in dispatchBlocks , isTimeUp isn't necessary. Overall the patch looks pretty good. I realized that it doesn't really cover the HDFS-10977 scenario. May I suggest we update the subject to "Enforce iteration time limit and slow block move timeout in Balancer"? It doesn't look easy to add a unit test and I'm OK in this case because the added logic is pretty straightforward. I'm happy to open a separate test JIRA though.
          Hide
          kihwal Kihwal Lee added a comment -

          Attaching the updated patch.

          Show
          kihwal Kihwal Lee added a comment - Attaching the updated patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 1s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 6m 39s trunk passed
          +1 compile 0m 44s trunk passed
          +1 checkstyle 0m 30s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 45s trunk passed
          +1 javadoc 0m 40s trunk passed
          +1 mvninstall 0m 54s the patch passed
          +1 compile 0m 48s the patch passed
          +1 javac 0m 48s the patch passed
          -0 checkstyle 0m 29s hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 477 unchanged - 1 fixed = 481 total (was 478)
          +1 mvnsite 0m 55s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 2m 3s the patch passed
          +1 javadoc 0m 39s the patch passed
          -1 unit 65m 0s hadoop-hdfs in the patch failed.
          -1 asflicense 0m 19s The patch generated 2 ASF License warnings.
          84m 10s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
            hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
            hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
            hadoop.hdfs.server.datanode.TestDataNodeLifeline
            hadoop.hdfs.server.balancer.TestBalancer
            hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer
            hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer
            hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
            hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-11015
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834648/HDFS-11015-2.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 2e64e560c25d 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 754cb4e
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17251/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/17251/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17251/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17251/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/17251/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17251/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 1s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 39s trunk passed +1 compile 0m 44s trunk passed +1 checkstyle 0m 30s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 45s trunk passed +1 javadoc 0m 40s trunk passed +1 mvninstall 0m 54s the patch passed +1 compile 0m 48s the patch passed +1 javac 0m 48s the patch passed -0 checkstyle 0m 29s hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 477 unchanged - 1 fixed = 481 total (was 478) +1 mvnsite 0m 55s the patch passed +1 mvneclipse 0m 10s the patch passed -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 3s the patch passed +1 javadoc 0m 39s the patch passed -1 unit 65m 0s hadoop-hdfs in the patch failed. -1 asflicense 0m 19s The patch generated 2 ASF License warnings. 84m 10s Reason Tests Failed junit tests hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes   hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS   hadoop.hdfs.server.datanode.TestDataNodeLifeline   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer   hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped   hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-11015 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834648/HDFS-11015-2.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 2e64e560c25d 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 754cb4e Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17251/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/17251/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17251/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17251/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/17251/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17251/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          kihwal Kihwal Lee added a comment -

          Heh. Looks like the unit tests are doing their job.

          Show
          kihwal Kihwal Lee added a comment - Heh. Looks like the unit tests are doing their job.
          Hide
          kihwal Kihwal Lee added a comment -

          Silly me. I put isIterationOver() instead of !isIterationOver() for the condition to continue. Fixed it in the new patch.

          Show
          kihwal Kihwal Lee added a comment - Silly me. I put isIterationOver() instead of !isIterationOver() for the condition to continue. Fixed it in the new patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 15s trunk passed
          +1 compile 0m 44s trunk passed
          +1 checkstyle 0m 32s trunk passed
          +1 mvnsite 0m 52s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 41s trunk passed
          +1 javadoc 0m 39s trunk passed
          +1 mvninstall 0m 45s the patch passed
          +1 compile 0m 42s the patch passed
          +1 javac 0m 42s the patch passed
          -0 checkstyle 0m 28s hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 476 unchanged - 1 fixed = 480 total (was 477)
          +1 mvnsite 0m 47s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 1m 45s the patch passed
          +1 javadoc 0m 36s the patch passed
          -1 unit 66m 19s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 27s The patch does not generate ASF License warnings.
          85m 18s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeUUID
            hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-11015
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835020/HDFS-11015-3.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 6501c4a90a8e 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a1a0281
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17266/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17266/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17266/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17266/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 15s trunk passed +1 compile 0m 44s trunk passed +1 checkstyle 0m 32s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 41s trunk passed +1 javadoc 0m 39s trunk passed +1 mvninstall 0m 45s the patch passed +1 compile 0m 42s the patch passed +1 javac 0m 42s the patch passed -0 checkstyle 0m 28s hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 476 unchanged - 1 fixed = 480 total (was 477) +1 mvnsite 0m 47s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 1m 45s the patch passed +1 javadoc 0m 36s the patch passed -1 unit 66m 19s hadoop-hdfs in the patch failed. +1 asflicense 0m 27s The patch does not generate ASF License warnings. 85m 18s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeUUID   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-11015 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835020/HDFS-11015-3.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 6501c4a90a8e 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a1a0281 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17266/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17266/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17266/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17266/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          kihwal Kihwal Lee added a comment -

          The failed tests all pass when run on my machine.

          Show
          kihwal Kihwal Lee added a comment - The failed tests all pass when run on my machine.
          Hide
          zhz Zhe Zhang added a comment - - edited

          +1 on the updated patch. Thanks Kihwal Lee.

          I just committed the patch to trunk. If you are OK with it I also plan to commit to branch-2 ~ branch-2.7. I also created HDFS-11051 to enhance testing around Balancer slow block moves.

          Show
          zhz Zhe Zhang added a comment - - edited +1 on the updated patch. Thanks Kihwal Lee . I just committed the patch to trunk. If you are OK with it I also plan to commit to branch-2 ~ branch-2.7. I also created HDFS-11051 to enhance testing around Balancer slow block moves.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10674 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10674/)
          HDFS-11015. Enforce timeout in balancer. Contributed by Kihwal Lee. (zhz: rev f6367c5f44a88cb5eb7edffb015b10b657504a61)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10674 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10674/ ) HDFS-11015 . Enforce timeout in balancer. Contributed by Kihwal Lee. (zhz: rev f6367c5f44a88cb5eb7edffb015b10b657504a61) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
          Hide
          zhz Zhe Zhang added a comment -

          Actually just noticed the original target version is 2.8. I committed to branch-2 and branch-2.8. Backporting to branch-2.7 is not very clean. I'm working on it.

          Show
          zhz Zhe Zhang added a comment - Actually just noticed the original target version is 2.8. I committed to branch-2 and branch-2.8. Backporting to branch-2.7 is not very clean. I'm working on it.
          Hide
          zhz Zhe Zhang added a comment -

          Committed to branch-2.7 as well. Thanks Kihwal Lee for the nice work!

          Show
          zhz Zhe Zhang added a comment - Committed to branch-2.7 as well. Thanks Kihwal Lee for the nice work!
          Hide
          kihwal Kihwal Lee added a comment -

          Zhe Zhang, thanks for reviews and commits.

          Show
          kihwal Kihwal Lee added a comment - Zhe Zhang , thanks for reviews and commits.

            People

            • Assignee:
              kihwal Kihwal Lee
              Reporter:
              kihwal Kihwal Lee
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development