Details

    • Hadoop Flags:
      Reviewed

      Description

      Balancer gets datanode block lists from NN and then move the blocks in order to balance the cluster. It should not use the blocks with small size since moving the small blocks generates a lot of overhead and the small blocks do not help balancing the cluster much.

      1. h8824_20150727b.patch
        21 kB
        Tsz Wo Nicholas Sze
      2. h8824_20150811b.patch
        10 kB
        Tsz Wo Nicholas Sze

        Issue Links

          Activity

          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          h8824_20150727b.patch: add minBlockSize to getBlocks(..).

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - h8824_20150727b.patch: add minBlockSize to getBlocks(..).
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 17s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 40s There were no new javac warning messages.
          +1 javadoc 9m 39s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 26s The applied patch generated 12 new checkstyle issues (total was 786, now 792).
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 22s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 2m 34s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 3s Pre-build of native portion
          -1 hdfs tests 221m 38s Tests failed in hadoop-hdfs.
              265m 39s  



          Reason Tests
          Failed unit tests hadoop.hdfs.TestFileStatus
            hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
            hadoop.hdfs.TestReadWhileWriting
            hadoop.hdfs.TestFSOutputSummer
            hadoop.hdfs.TestParallelShortCircuitLegacyRead
            hadoop.hdfs.TestAppendSnapshotTruncate
            hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer
            hadoop.hdfs.TestDFSClientRetries
            hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
          Timed out tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
            org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
            org.apache.hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer
            org.apache.hadoop.hdfs.server.balancer.TestBalancer



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12747457/h8824_20150727b.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 3e6fce9
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11850/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11850/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11850/testReport/
          Java 1.7.0_55
          uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11850/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 17s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 40s There were no new javac warning messages. +1 javadoc 9m 39s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 26s The applied patch generated 12 new checkstyle issues (total was 786, now 792). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 22s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 2m 34s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 3s Pre-build of native portion -1 hdfs tests 221m 38s Tests failed in hadoop-hdfs.     265m 39s   Reason Tests Failed unit tests hadoop.hdfs.TestFileStatus   hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes   hadoop.hdfs.TestReadWhileWriting   hadoop.hdfs.TestFSOutputSummer   hadoop.hdfs.TestParallelShortCircuitLegacyRead   hadoop.hdfs.TestAppendSnapshotTruncate   hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer   hadoop.hdfs.TestDFSClientRetries   hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup Timed out tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS   org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes   org.apache.hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer   org.apache.hadoop.hdfs.server.balancer.TestBalancer Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12747457/h8824_20150727b.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 3e6fce9 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11850/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11850/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11850/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11850/console This message was automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          h8824_20150811b.patch: reverts the NN change. Will do it in a separated JIRA.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - h8824_20150811b.patch: reverts the NN change. Will do it in a separated JIRA.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 27s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 44s There were no new javac warning messages.
          +1 javadoc 9m 52s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 23s The applied patch generated 5 new checkstyle issues (total was 523, now 525).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 20s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 2m 31s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 7s Pre-build of native portion
          -1 hdfs tests 90m 5s Tests failed in hadoop-hdfs.
              134m 31s  



          Reason Tests
          Failed unit tests hadoop.hdfs.TestAppendSnapshotTruncate
            hadoop.hdfs.TestLeaseRecovery2
          Timed out tests org.apache.hadoop.hdfs.TestListFilesInFileContext
            org.apache.hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12749952/h8824_20150811b.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 7c796fd
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11969/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11969/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11969/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11969/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 27s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 44s There were no new javac warning messages. +1 javadoc 9m 52s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 23s The applied patch generated 5 new checkstyle issues (total was 523, now 525). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 20s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 2m 31s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 7s Pre-build of native portion -1 hdfs tests 90m 5s Tests failed in hadoop-hdfs.     134m 31s   Reason Tests Failed unit tests hadoop.hdfs.TestAppendSnapshotTruncate   hadoop.hdfs.TestLeaseRecovery2 Timed out tests org.apache.hadoop.hdfs.TestListFilesInFileContext   org.apache.hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12749952/h8824_20150811b.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 7c796fd checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11969/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11969/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11969/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11969/console This message was automatically generated.
          Hide
          jnp Jitendra Nath Pandey added a comment -

          +1 for the latest patch.

          Show
          jnp Jitendra Nath Pandey added a comment - +1 for the latest patch.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8308 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8308/)
          HDFS-8824. Do not use small blocks for balancing the cluster. (szetszwo: rev 2bc0a4f299fbd8035e29f62ce9cd22e209a62805)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8308 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8308/ ) HDFS-8824 . Do not use small blocks for balancing the cluster. (szetszwo: rev 2bc0a4f299fbd8035e29f62ce9cd22e209a62805) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Thanks Jitendra for reviewing the patch.

          I have committed this.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Thanks Jitendra for reviewing the patch. I have committed this.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #288 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/288/)
          HDFS-8824. Do not use small blocks for balancing the cluster. (szetszwo: rev 2bc0a4f299fbd8035e29f62ce9cd22e209a62805)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #288 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/288/ ) HDFS-8824 . Do not use small blocks for balancing the cluster. (szetszwo: rev 2bc0a4f299fbd8035e29f62ce9cd22e209a62805) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1018 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1018/)
          HDFS-8824. Do not use small blocks for balancing the cluster. (szetszwo: rev 2bc0a4f299fbd8035e29f62ce9cd22e209a62805)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1018 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1018/ ) HDFS-8824 . Do not use small blocks for balancing the cluster. (szetszwo: rev 2bc0a4f299fbd8035e29f62ce9cd22e209a62805) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2234 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2234/)
          HDFS-8824. Do not use small blocks for balancing the cluster. (szetszwo: rev 2bc0a4f299fbd8035e29f62ce9cd22e209a62805)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2234 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2234/ ) HDFS-8824 . Do not use small blocks for balancing the cluster. (szetszwo: rev 2bc0a4f299fbd8035e29f62ce9cd22e209a62805) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #285 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/285/)
          HDFS-8824. Do not use small blocks for balancing the cluster. (szetszwo: rev 2bc0a4f299fbd8035e29f62ce9cd22e209a62805)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #285 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/285/ ) HDFS-8824 . Do not use small blocks for balancing the cluster. (szetszwo: rev 2bc0a4f299fbd8035e29f62ce9cd22e209a62805) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2215 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2215/)
          HDFS-8824. Do not use small blocks for balancing the cluster. (szetszwo: rev 2bc0a4f299fbd8035e29f62ce9cd22e209a62805)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2215 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2215/ ) HDFS-8824 . Do not use small blocks for balancing the cluster. (szetszwo: rev 2bc0a4f299fbd8035e29f62ce9cd22e209a62805) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #277 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/277/)
          HDFS-8824. Do not use small blocks for balancing the cluster. (szetszwo: rev 2bc0a4f299fbd8035e29f62ce9cd22e209a62805)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #277 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/277/ ) HDFS-8824 . Do not use small blocks for balancing the cluster. (szetszwo: rev 2bc0a4f299fbd8035e29f62ce9cd22e209a62805) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Nicholas for the work. I think this is a good improvement for branch-2.7 and just backported. Verified TestBalancer.

          Show
          zhz Zhe Zhang added a comment - Thanks Nicholas for the work. I think this is a good improvement for branch-2.7 and just backported. Verified TestBalancer .
          Hide
          kihwal Kihwal Lee added a comment -

          While this will initially increase the efficiency of balancing, it is not without a negative side-effect.

          Older nodes in a cluster will slowly filled with smaller blocks as time goes on. This is accelerated if the cluster is heterogeneous. The smaller nodes will fill up more quickly/frequently and the balancer will move only big blocks out of those nodes. As more balacing happens, those nodes will contain more and more small blocks. If sufficient time passes, the blocks on those nodes will almost entirely small.

          This feature can be enabled for quickly resolving a storage balance issue, but long-term use can have unintended side-effect. Fortunately, we have not released any (other than alpha) with this feature. We can include more information in the release note and/or address the issue in the code/config.

          Show
          kihwal Kihwal Lee added a comment - While this will initially increase the efficiency of balancing, it is not without a negative side-effect. Older nodes in a cluster will slowly filled with smaller blocks as time goes on. This is accelerated if the cluster is heterogeneous. The smaller nodes will fill up more quickly/frequently and the balancer will move only big blocks out of those nodes. As more balacing happens, those nodes will contain more and more small blocks. If sufficient time passes, the blocks on those nodes will almost entirely small. This feature can be enabled for quickly resolving a storage balance issue, but long-term use can have unintended side-effect. Fortunately, we have not released any (other than alpha) with this feature. We can include more information in the release note and/or address the issue in the code/config.
          Hide
          zhz Zhe Zhang added a comment -

          Good finding Kihwal. +1 on extending the documentation with this effect.

          Should we change the default value of 10MB?

          Show
          zhz Zhe Zhang added a comment - Good finding Kihwal. +1 on extending the documentation with this effect. Should we change the default value of 10MB?

            People

            • Assignee:
              szetszwo Tsz Wo Nicholas Sze
              Reporter:
              szetszwo Tsz Wo Nicholas Sze
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development