Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8131

Implement a space balanced block placement policy

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The default block placement policy will choose datanodes for new blocks randomly, which will result in unbalanced space used percent among datanodes after an cluster expansion. The old datanodes always are in high used percent of space and new added ones are in low percent.

      Through we can used the external balance tool to balance the space used rate, it will cost extra network IO and it's not easy to control the balance speed.

      An easy solution is to implement an balanced block placement policy which will choose low used percent datanodes for new blocks with a little high possibility. In a not long term, the used percent of datanodes will trend to be balanced.

      Suggestions and discussions are welcomed. Thanks

      1. balanced.png
        40 kB
        Liu Shaohui
      2. HDFS-8131.004.patch
        14 kB
        Liu Shaohui
      3. HDFS-8131.005.patch
        14 kB
        Liu Shaohui
      4. HDFS-8131.006.patch
        14 kB
        Liu Shaohui
      5. HDFS-8131-v1.diff
        14 kB
        Liu Shaohui
      6. HDFS-8131-v2.diff
        14 kB
        Liu Shaohui
      7. HDFS-8131-v3.diff
        14 kB
        Liu Shaohui

        Issue Links

          Activity

          Hide
          liushaohui Liu Shaohui added a comment -

          Patch for trunk

          Show
          liushaohui Liu Shaohui added a comment - Patch for trunk
          Hide
          walter.k.su Walter Su added a comment -

          Interesting! The logic looks good. Have you test it on a cluster? A suggestion, integrate it into default policy, and add a switch at configuration.

          Show
          walter.k.su Walter Su added a comment - Interesting! The logic looks good. Have you test it on a cluster? A suggestion, integrate it into default policy, and add a switch at configuration.
          Hide
          liushaohui Liu Shaohui added a comment -

          Walter Su

          Have you test it on a cluster?

          Tests in a cluster are ongoing. I will update results later.

          A suggestion, integrate it into default policy, and add a switch at configuration

          I thought a plugin may be more acceptable. If the community prefer this, I will integrate it into default policy.

          Show
          liushaohui Liu Shaohui added a comment - Walter Su Have you test it on a cluster? Tests in a cluster are ongoing. I will update results later. A suggestion, integrate it into default policy, and add a switch at configuration I thought a plugin may be more acceptable. If the community prefer this, I will integrate it into default policy.
          Hide
          knoguchi Koji Noguchi added a comment -

          Hi Liu. Have you taken a look at Kihwal Lee's HDFS-8041?

          In general, recently generated data are more likely to get accessed than the old ones. In that sense, we don't want all the new blocks to be copied to the newly added nodes since those can quickly become the bottleneck.

          I think HDFS-8041 hits a good balance.

          Show
          knoguchi Koji Noguchi added a comment - Hi Liu. Have you taken a look at Kihwal Lee 's HDFS-8041 ? In general, recently generated data are more likely to get accessed than the old ones. In that sense, we don't want all the new blocks to be copied to the newly added nodes since those can quickly become the bottleneck. I think HDFS-8041 hits a good balance.
          Hide
          liushaohui Liu Shaohui added a comment -

          Koji Noguchi

          In that sense, we don't want all the new blocks to be copied to the newly added nodes since those can quickly become the bottleneck.

          The new added nodes are chosen with a little high possibility than the old ones. Please look at the patch.
          If there are p percent new added nodes in the cluster. In the default block placement policy, the new added nodes will be chosen with possibility: p. In the balanced block placement policy in this patch, the possibility will be q = p * p + 2* (1-p) * p * 0.6. (0.6 is configurable)
          if p = 0.5, q = 0.55. if p = 0.01, q = 0.01198. The percent of increased possibility is not very exceeding。

          I think HDFS-8041 hits a good balance.

          I think there are differences between the two patches.
          HDFS-8041 blocks the datanodes to be chosen if the space used percent is large than a threshold when the cluster is reasonably full.
          At that time, I think it's maybe to late and the cluster maybe lose a half of write capability if old nodes can not be chosen .

          The aim of this patch is to make the space balanced in a long term. The new added nodes are chosen with a little high possibility than the old ones. It's a very smooth method.

          Show
          liushaohui Liu Shaohui added a comment - Koji Noguchi In that sense, we don't want all the new blocks to be copied to the newly added nodes since those can quickly become the bottleneck. The new added nodes are chosen with a little high possibility than the old ones. Please look at the patch. If there are p percent new added nodes in the cluster. In the default block placement policy, the new added nodes will be chosen with possibility: p. In the balanced block placement policy in this patch, the possibility will be q = p * p + 2* (1-p) * p * 0.6. (0.6 is configurable) if p = 0.5, q = 0.55. if p = 0.01, q = 0.01198. The percent of increased possibility is not very exceeding。 I think HDFS-8041 hits a good balance. I think there are differences between the two patches. HDFS-8041 blocks the datanodes to be chosen if the space used percent is large than a threshold when the cluster is reasonably full. At that time, I think it's maybe to late and the cluster maybe lose a half of write capability if old nodes can not be chosen . The aim of this patch is to make the space balanced in a long term. The new added nodes are chosen with a little high possibility than the old ones. It's a very smooth method.
          Hide
          liushaohui Liu Shaohui added a comment -

          Kihwal Lee
          Please help to point out if my understanding is wrong about HDFS-8041 in last comment.

          Show
          liushaohui Liu Shaohui added a comment - Kihwal Lee Please help to point out if my understanding is wrong about HDFS-8041 in last comment.
          Hide
          knoguchi Koji Noguchi added a comment -

          Thanks for correcting my misunderstanding.
          I clearly missed the "a little high possibility" part. My bad.

          Show
          knoguchi Koji Noguchi added a comment - Thanks for correcting my misunderstanding. I clearly missed the "a little high possibility" part. My bad.
          Hide
          liushaohui Liu Shaohui added a comment -

          Never mind. Would you help to review the patch?
          Suggestions and discussions are welcomed.

          Show
          liushaohui Liu Shaohui added a comment - Never mind. Would you help to review the patch? Suggestions and discussions are welcomed.
          Hide
          liushaohui Liu Shaohui added a comment -

          Update:Test in a cluster and the BBP works as expected.
          Test method:
          a. Write data to a cluster with 3 data nodes until the disk used percent to 20
          b, Add two data nodes to the cluster and write data to the cluster continually.
          The disk used percent of all 5 data nodes will balance in a long term.

          See: the graph in the attachment

          Show
          liushaohui Liu Shaohui added a comment - Update:Test in a cluster and the BBP works as expected. Test method: a. Write data to a cluster with 3 data nodes until the disk used percent to 20 b, Add two data nodes to the cluster and write data to the cluster continually. The disk used percent of all 5 data nodes will balance in a long term. See: the graph in the attachment
          Hide
          liushaohui Liu Shaohui added a comment -

          Update the patch.

          Show
          liushaohui Liu Shaohui added a comment - Update the patch.
          Hide
          liushaohui Liu Shaohui added a comment -

          Could someone help to review this patch? Thanks

          Show
          liushaohui Liu Shaohui added a comment - Could someone help to review this patch? Thanks
          Hide
          liushaohui Liu Shaohui added a comment -

          Rebase on trunk

          Show
          liushaohui Liu Shaohui added a comment - Rebase on trunk
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 patch 0m 1s The patch file was not named according to hadoop's naming conventions. Please see https://wiki.apache.org/hadoop/HowToContribute for instructions.
          0 pre-patch 14m 34s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 29s There were no new javac warning messages.
          -1 javadoc 9m 39s The applied patch generated 1 additional warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 15s The applied patch generated 16 new checkstyle issues (total was 459, now 475).
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 32s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 3m 4s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 native 3m 14s Pre-build of native portion
          -1 hdfs tests 168m 5s Tests failed in hadoop-hdfs.
              210m 51s  



          Reason Tests
          Failed unit tests hadoop.tools.TestHdfsConfigFields



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12732816/HDFS-8131-v3.diff
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 0daa5ad
          javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/10972/artifact/patchprocess/diffJavadocWarnings.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10972/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/10972/artifact/patchprocess/whitespace.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10972/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10972/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10972/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 patch 0m 1s The patch file was not named according to hadoop's naming conventions. Please see https://wiki.apache.org/hadoop/HowToContribute for instructions. 0 pre-patch 14m 34s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 29s There were no new javac warning messages. -1 javadoc 9m 39s The applied patch generated 1 additional warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 15s The applied patch generated 16 new checkstyle issues (total was 459, now 475). -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 4s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 native 3m 14s Pre-build of native portion -1 hdfs tests 168m 5s Tests failed in hadoop-hdfs.     210m 51s   Reason Tests Failed unit tests hadoop.tools.TestHdfsConfigFields Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12732816/HDFS-8131-v3.diff Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 0daa5ad javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/10972/artifact/patchprocess/diffJavadocWarnings.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10972/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/10972/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10972/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10972/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10972/console This message was automatically generated.
          Hide
          liushaohui Liu Shaohui added a comment -

          The failed test: TestHdfsConfigFields has no relations with this patch. It failed in trunk also.

          Show
          liushaohui Liu Shaohui added a comment - The failed test: TestHdfsConfigFields has no relations with this patch. It failed in trunk also.
          Hide
          liushaohui Liu Shaohui added a comment -

          Fix problems of javadoc and remove while space

          Show
          liushaohui Liu Shaohui added a comment - Fix problems of javadoc and remove while space
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 57s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 42s There were no new javac warning messages.
          +1 javadoc 9m 53s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 14s The applied patch generated 15 new checkstyle issues (total was 459, now 474).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 35s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 3m 4s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 native 3m 26s Pre-build of native portion
          -1 hdfs tests 184m 1s Tests failed in hadoop-hdfs.
              227m 52s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.blockmanagement.TestDatanodeManager
            hadoop.tools.TestHdfsConfigFields



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12732849/HDFS-8131.004.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / def9136
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10979/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10979/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10979/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10979/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 57s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 42s There were no new javac warning messages. +1 javadoc 9m 53s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 14s The applied patch generated 15 new checkstyle issues (total was 459, now 474). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 3m 4s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 native 3m 26s Pre-build of native portion -1 hdfs tests 184m 1s Tests failed in hadoop-hdfs.     227m 52s   Reason Tests Failed unit tests hadoop.hdfs.server.blockmanagement.TestDatanodeManager   hadoop.tools.TestHdfsConfigFields Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12732849/HDFS-8131.004.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / def9136 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/10979/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10979/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10979/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10979/console This message was automatically generated.
          Hide
          liushaohui Liu Shaohui added a comment -

          Failed tests have no relations with this patch.

          Show
          liushaohui Liu Shaohui added a comment - Failed tests have no relations with this patch.
          Hide
          kihwal Kihwal Lee added a comment -

          Liu Shaohui Sorry for the dealy. I will try to review it today.

          Show
          kihwal Kihwal Lee added a comment - Liu Shaohui Sorry for the dealy. I will try to review it today.
          Hide
          kihwal Kihwal Lee added a comment -

          The overall approach and organization seems reasonable. Can you change the internal operations to integer-based? E.g. integer percent.

          Show
          kihwal Kihwal Lee added a comment - The overall approach and organization seems reasonable. Can you change the internal operations to integer-based? E.g. integer percent.
          Hide
          liushaohui Liu Shaohui added a comment -

          Update for Kihwal Lee's review.
          Thanks

          Show
          liushaohui Liu Shaohui added a comment - Update for Kihwal Lee 's review. Thanks
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 43s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 30s There were no new javac warning messages.
          +1 javadoc 9m 36s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 13s The applied patch generated 14 new checkstyle issues (total was 460, now 474).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 3m 6s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 native 3m 13s Pre-build of native portion
          +1 hdfs tests 163m 32s Tests passed in hadoop-hdfs.
              206m 26s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12733437/HDFS-8131.005.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / cab0dad
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11022/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11022/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11022/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11022/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 43s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 30s There were no new javac warning messages. +1 javadoc 9m 36s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 13s The applied patch generated 14 new checkstyle issues (total was 460, now 474). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 3m 6s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 native 3m 13s Pre-build of native portion +1 hdfs tests 163m 32s Tests passed in hadoop-hdfs.     206m 26s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12733437/HDFS-8131.005.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / cab0dad checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11022/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11022/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11022/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11022/console This message was automatically generated.
          Hide
          kihwal Kihwal Lee added a comment -

          The checkstyle warning can be ignored for the lines with the long variable names. I think you can fix line 35, 36, 62, 84 and 85. Other than that the code change looks good.

          Show
          kihwal Kihwal Lee added a comment - The checkstyle warning can be ignored for the lines with the long variable names. I think you can fix line 35, 36, 62, 84 and 85. Other than that the code change looks good.
          Hide
          liushaohui Liu Shaohui added a comment -

          Update for Kihwal Lee's review.

          • Format the long lines.
          Show
          liushaohui Liu Shaohui added a comment - Update for Kihwal Lee 's review. Format the long lines.
          Hide
          liushaohui Liu Shaohui added a comment -

          Thanks for Kihwal Lee's review.

          Show
          liushaohui Liu Shaohui added a comment - Thanks for Kihwal Lee 's review.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 15m 8s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 41s There were no new javac warning messages.
          +1 javadoc 9m 56s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 38s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 40s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 3m 21s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 18s Pre-build of native portion
          +1 hdfs tests 167m 57s Tests passed in hadoop-hdfs.
              210m 38s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12733696/HDFS-8131.006.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 0790275
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11042/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11042/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11042/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 15m 8s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 41s There were no new javac warning messages. +1 javadoc 9m 56s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 38s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 40s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 21s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 18s Pre-build of native portion +1 hdfs tests 167m 57s Tests passed in hadoop-hdfs.     210m 38s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12733696/HDFS-8131.006.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 0790275 hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11042/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11042/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11042/console This message was automatically generated.
          Hide
          kihwal Kihwal Lee added a comment -

          +1 the latest patch looks good.

          Show
          kihwal Kihwal Lee added a comment - +1 the latest patch looks good.
          Hide
          kihwal Kihwal Lee added a comment -

          I've committed this to trunk and branch-2. Thanks for working on this, Liu Shaohui.

          Show
          kihwal Kihwal Lee added a comment - I've committed this to trunk and branch-2. Thanks for working on this, Liu Shaohui .
          Hide
          walter.k.su Walter Su added a comment -

          Hi, Kihwal Lee! Could you take a look at HDFS-8390? It's related. Do you think it's necessary or not?

          Show
          walter.k.su Walter Su added a comment - Hi, Kihwal Lee ! Could you take a look at HDFS-8390 ? It's related. Do you think it's necessary or not?
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7861 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7861/)
          HDFS-8131. Implement a space balanced block placement policy. Contributed by Liu Shaohui. (kihwal: rev de30d66b2673d0344346fb985e786247ca682317)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/AvailableSpaceBlockPlacementPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestAvailableSpaceBlockPlacementPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7861 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7861/ ) HDFS-8131 . Implement a space balanced block placement policy. Contributed by Liu Shaohui. (kihwal: rev de30d66b2673d0344346fb985e786247ca682317) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/AvailableSpaceBlockPlacementPolicy.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestAvailableSpaceBlockPlacementPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          Hide
          liushaohui Liu Shaohui added a comment -

          Thanks~ Kihwal Lee

          Show
          liushaohui Liu Shaohui added a comment - Thanks~ Kihwal Lee
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #202 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/202/)
          HDFS-8131. Implement a space balanced block placement policy. Contributed by Liu Shaohui. (kihwal: rev de30d66b2673d0344346fb985e786247ca682317)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestAvailableSpaceBlockPlacementPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/AvailableSpaceBlockPlacementPolicy.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #202 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/202/ ) HDFS-8131 . Implement a space balanced block placement policy. Contributed by Liu Shaohui. (kihwal: rev de30d66b2673d0344346fb985e786247ca682317) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestAvailableSpaceBlockPlacementPolicy.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/AvailableSpaceBlockPlacementPolicy.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #933 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/933/)
          HDFS-8131. Implement a space balanced block placement policy. Contributed by Liu Shaohui. (kihwal: rev de30d66b2673d0344346fb985e786247ca682317)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/AvailableSpaceBlockPlacementPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestAvailableSpaceBlockPlacementPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #933 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/933/ ) HDFS-8131 . Implement a space balanced block placement policy. Contributed by Liu Shaohui. (kihwal: rev de30d66b2673d0344346fb985e786247ca682317) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/AvailableSpaceBlockPlacementPolicy.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestAvailableSpaceBlockPlacementPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2131 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2131/)
          HDFS-8131. Implement a space balanced block placement policy. Contributed by Liu Shaohui. (kihwal: rev de30d66b2673d0344346fb985e786247ca682317)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestAvailableSpaceBlockPlacementPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/AvailableSpaceBlockPlacementPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2131 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2131/ ) HDFS-8131 . Implement a space balanced block placement policy. Contributed by Liu Shaohui. (kihwal: rev de30d66b2673d0344346fb985e786247ca682317) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestAvailableSpaceBlockPlacementPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/AvailableSpaceBlockPlacementPolicy.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #191 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/191/)
          HDFS-8131. Implement a space balanced block placement policy. Contributed by Liu Shaohui. (kihwal: rev de30d66b2673d0344346fb985e786247ca682317)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestAvailableSpaceBlockPlacementPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/AvailableSpaceBlockPlacementPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #191 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/191/ ) HDFS-8131 . Implement a space balanced block placement policy. Contributed by Liu Shaohui. (kihwal: rev de30d66b2673d0344346fb985e786247ca682317) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestAvailableSpaceBlockPlacementPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/AvailableSpaceBlockPlacementPolicy.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #201 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/201/)
          HDFS-8131. Implement a space balanced block placement policy. Contributed by Liu Shaohui. (kihwal: rev de30d66b2673d0344346fb985e786247ca682317)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/AvailableSpaceBlockPlacementPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestAvailableSpaceBlockPlacementPolicy.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #201 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/201/ ) HDFS-8131 . Implement a space balanced block placement policy. Contributed by Liu Shaohui. (kihwal: rev de30d66b2673d0344346fb985e786247ca682317) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/AvailableSpaceBlockPlacementPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestAvailableSpaceBlockPlacementPolicy.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2149 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2149/)
          HDFS-8131. Implement a space balanced block placement policy. Contributed by Liu Shaohui. (kihwal: rev de30d66b2673d0344346fb985e786247ca682317)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/AvailableSpaceBlockPlacementPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestAvailableSpaceBlockPlacementPolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2149 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2149/ ) HDFS-8131 . Implement a space balanced block placement policy. Contributed by Liu Shaohui. (kihwal: rev de30d66b2673d0344346fb985e786247ca682317) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/AvailableSpaceBlockPlacementPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestAvailableSpaceBlockPlacementPolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
          Hide
          zhz Zhe Zhang added a comment -

          AvailableSpaceBlockPlacementPolicy is very useful for large production clusters, many of which are still running 2.6/2.7. So if there are no objections I plan to backport this to branch-2.7. Will wait until end of Wednesday.

          We are still discussing under HDFS-10967 whether to merge AvailableSpaceBlockPlacementPolicy into the default BPP. But I think whatever we decide, can be done on top of this one.

          Show
          zhz Zhe Zhang added a comment - AvailableSpaceBlockPlacementPolicy is very useful for large production clusters, many of which are still running 2.6/2.7. So if there are no objections I plan to backport this to branch-2.7. Will wait until end of Wednesday. We are still discussing under HDFS-10967 whether to merge AvailableSpaceBlockPlacementPolicy into the default BPP. But I think whatever we decide, can be done on top of this one.
          Hide
          Tagar Ruslan Dautkhanov added a comment -

          Thanks for this great improvement!
          When using AvailableSpaceBlockPlacementPolicy, the default below logic does not work anymore?

          1. Place the first replica somewhere – either a random rack and node (if the HDFS client is outside the hadoop cluster) or on the local node (if the HDFS client is running on a node inside the cluster).
          2. The second replica is written to a different rack from the first, chosen at random.
          3. The third replica is written to the same rack as the second replica, but on a different node.
          4. If there are more replicas – spread them across the rest of the racks.

          What is this logic now? When it comes to rackawareness and such?
          Is it by pure available space and rack awareness logic doesn't kick in?

          Show
          Tagar Ruslan Dautkhanov added a comment - Thanks for this great improvement! When using AvailableSpaceBlockPlacementPolicy, the default below logic does not work anymore? 1. Place the first replica somewhere – either a random rack and node (if the HDFS client is outside the hadoop cluster) or on the local node (if the HDFS client is running on a node inside the cluster). 2. The second replica is written to a different rack from the first, chosen at random. 3. The third replica is written to the same rack as the second replica, but on a different node. 4. If there are more replicas – spread them across the rest of the racks. What is this logic now? When it comes to rackawareness and such? Is it by pure available space and rack awareness logic doesn't kick in?
          Hide
          maobaolong maobaolong added a comment -

          Liu Shaohui
          Hello, Thanks for this great improvement!
          I have an question about the config key DFS_NAMENODE_AVAILABLE_SPACE_BLOCK_PLACEMENT_POLICY_BALANCED_SPACE_PREFERENCE_FRACTION_DEFAULT.
          What is the situation when this key is set between 0.5 and 1.0, i think it is better to set to 1.0 always, Is that true? There must be something i don't understand well, Please point it, thank you advance.

          Show
          maobaolong maobaolong added a comment - Liu Shaohui Hello, Thanks for this great improvement! I have an question about the config key DFS_NAMENODE_AVAILABLE_SPACE_BLOCK_PLACEMENT_POLICY_BALANCED_SPACE_PREFERENCE_FRACTION_DEFAULT. What is the situation when this key is set between 0.5 and 1.0, i think it is better to set to 1.0 always, Is that true? There must be something i don't understand well, Please point it, thank you advance.

            People

            • Assignee:
              liushaohui Liu Shaohui
              Reporter:
              liushaohui Liu Shaohui
            • Votes:
              0 Vote for this issue
              Watchers:
              21 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development