Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: balancer & mover
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Balancer is inefficient in the following case:

      Datanode Utilization Rack
      D1 95% A
      D2 30% B
      D3, D4, D5 0% B

      The average utilization is 25% so that D2 is within 10% threshold. However, Balancer currently will first move blocks from D2 to D3, D4 and D5 since they are under the same rack. Then, it will move blocks from D1.

      1. h8826_20150811.patch
        27 kB
        Tsz Wo Nicholas Sze
      2. h8826_20150816.patch
        29 kB
        Tsz Wo Nicholas Sze
      3. h8826_20150818.patch
        29 kB
        Tsz Wo Nicholas Sze

        Issue Links

          Activity

          Hide
          ctrezzo Chris Trezzo added a comment -

          Woops. Meant HDFS-8923.

          Show
          ctrezzo Chris Trezzo added a comment - Woops. Meant HDFS-8923 .
          Hide
          ctrezzo Chris Trezzo added a comment -

          HDFS-8826 filled. Posting patch there.

          Show
          ctrezzo Chris Trezzo added a comment - HDFS-8826 filled. Posting patch there.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #289 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/289/)
          HDFS-8826. In Balancer, add an option to specify the source node list so that balancer only selects blocks to move from those nodes. (szetszwo: rev 7ecbfd44aa57f5f54c214b7fdedda2500be76f51)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HostsFileReader.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #289 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/289/ ) HDFS-8826 . In Balancer, add an option to specify the source node list so that balancer only selects blocks to move from those nodes. (szetszwo: rev 7ecbfd44aa57f5f54c214b7fdedda2500be76f51) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HostsFileReader.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
          Hide
          ctrezzo Chris Trezzo added a comment -

          Arpit Agarwal Tsz Wo Nicholas Sze

          The applied patch is currently missing the -source flag in the usage message in the balancer. See current usage message in trunk:

             private static final String USAGE = "Usage: hdfs balancer"
                 + "\n\t[-policy <policy>]\tthe balancing policy: "
                 + BalancingPolicy.Node.INSTANCE.getName() + " or "
                 + BalancingPolicy.Pool.INSTANCE.getName()
                 + "\n\t[-threshold <threshold>]\tPercentage of disk capacity"
                 + "\n\t[-exclude [-f <hosts-file> | <comma-separated list of hosts>]]"
                 + "\tExcludes the specified datanodes."
                 + "\n\t[-include [-f <hosts-file> | <comma-separated list of hosts>]]"
                 + "\tIncludes only the specified datanodes."
                 + "\n\t[-idleiterations <idleiterations>]"
                 + "\tNumber of consecutive idle iterations (-1 for Infinite) before "
                 + "exit."
                 + "\n\t[-runDuringUpgrade]"
                 + "\tWhether to run the balancer during an ongoing HDFS upgrade."
                 + "This is usually not desired since it will not affect used space "
                 + "on over-utilized machines.";
          

          Should I file a jira or do you guys just want to post an amendment patch? Thanks!

          Show
          ctrezzo Chris Trezzo added a comment - Arpit Agarwal Tsz Wo Nicholas Sze The applied patch is currently missing the -source flag in the usage message in the balancer. See current usage message in trunk: private static final String USAGE = "Usage: hdfs balancer" + "\n\t[-policy <policy>]\tthe balancing policy: " + BalancingPolicy.Node.INSTANCE.getName() + " or " + BalancingPolicy.Pool.INSTANCE.getName() + "\n\t[-threshold <threshold>]\tPercentage of disk capacity" + "\n\t[-exclude [-f <hosts-file> | <comma-separated list of hosts>]]" + "\tExcludes the specified datanodes." + "\n\t[-include [-f <hosts-file> | <comma-separated list of hosts>]]" + "\tIncludes only the specified datanodes." + "\n\t[-idleiterations <idleiterations>]" + "\tNumber of consecutive idle iterations (-1 for Infinite) before " + "exit." + "\n\t[-runDuringUpgrade]" + "\tWhether to run the balancer during an ongoing HDFS upgrade." + "This is usually not desired since it will not affect used space " + "on over-utilized machines." ; Should I file a jira or do you guys just want to post an amendment patch? Thanks!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2238 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2238/)
          HDFS-8826. In Balancer, add an option to specify the source node list so that balancer only selects blocks to move from those nodes. (szetszwo: rev 7ecbfd44aa57f5f54c214b7fdedda2500be76f51)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HostsFileReader.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2238 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2238/ ) HDFS-8826 . In Balancer, add an option to specify the source node list so that balancer only selects blocks to move from those nodes. (szetszwo: rev 7ecbfd44aa57f5f54c214b7fdedda2500be76f51) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HostsFileReader.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #281 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/281/)
          HDFS-8826. In Balancer, add an option to specify the source node list so that balancer only selects blocks to move from those nodes. (szetszwo: rev 7ecbfd44aa57f5f54c214b7fdedda2500be76f51)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HostsFileReader.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #281 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/281/ ) HDFS-8826 . In Balancer, add an option to specify the source node list so that balancer only selects blocks to move from those nodes. (szetszwo: rev 7ecbfd44aa57f5f54c214b7fdedda2500be76f51) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HostsFileReader.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2219 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2219/)
          HDFS-8826. In Balancer, add an option to specify the source node list so that balancer only selects blocks to move from those nodes. (szetszwo: rev 7ecbfd44aa57f5f54c214b7fdedda2500be76f51)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HostsFileReader.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2219 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2219/ ) HDFS-8826 . In Balancer, add an option to specify the source node list so that balancer only selects blocks to move from those nodes. (szetszwo: rev 7ecbfd44aa57f5f54c214b7fdedda2500be76f51) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HostsFileReader.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #292 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/292/)
          HDFS-8826. In Balancer, add an option to specify the source node list so that balancer only selects blocks to move from those nodes. (szetszwo: rev 7ecbfd44aa57f5f54c214b7fdedda2500be76f51)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HostsFileReader.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #292 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/292/ ) HDFS-8826 . In Balancer, add an option to specify the source node list so that balancer only selects blocks to move from those nodes. (szetszwo: rev 7ecbfd44aa57f5f54c214b7fdedda2500be76f51) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HostsFileReader.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1022 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1022/)
          HDFS-8826. In Balancer, add an option to specify the source node list so that balancer only selects blocks to move from those nodes. (szetszwo: rev 7ecbfd44aa57f5f54c214b7fdedda2500be76f51)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HostsFileReader.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1022 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1022/ ) HDFS-8826 . In Balancer, add an option to specify the source node list so that balancer only selects blocks to move from those nodes. (szetszwo: rev 7ecbfd44aa57f5f54c214b7fdedda2500be76f51) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HostsFileReader.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          > Also it may be a good idea to add a separate option to source from the most over-utilized DataNodes first so the administrator does not have to pass the source DNs manually ...

          It is a good idea. I filed HDFS-8921.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - > Also it may be a good idea to add a separate option to source from the most over-utilized DataNodes first so the administrator does not have to pass the source DNs manually ... It is a good idea. I filed HDFS-8921 .
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8319 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8319/)
          HDFS-8826. In Balancer, add an option to specify the source node list so that balancer only selects blocks to move from those nodes. (szetszwo: rev 7ecbfd44aa57f5f54c214b7fdedda2500be76f51)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HostsFileReader.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8319 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8319/ ) HDFS-8826 . In Balancer, add an option to specify the source node list so that balancer only selects blocks to move from those nodes. (szetszwo: rev 7ecbfd44aa57f5f54c214b7fdedda2500be76f51) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HostsFileReader.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Thanks Arpit for reviewing the patch.

          I have committed this.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Thanks Arpit for reviewing the patch. I have committed this.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          The new patch only has minor changes javadoc. I will commit it without waiting for Jenkins.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - The new patch only has minor changes javadoc. I will commit it without waiting for Jenkins.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          h8826_20150818.patch: adds periods at the end of the sentences in javadoc.

          I did not fix checkstyle comments like "Variable 'excludedNodes' must be private and have accessor methods." since it won't be consistent with the current code.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - h8826_20150818.patch: adds periods at the end of the sentences in javadoc. I did not fix checkstyle comments like "Variable 'excludedNodes' must be private and have accessor methods." since it won't be consistent with the current code.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Also it may be a good idea to add a separate option to source from the most over-utilized DataNodes first so the administrator does not have to pass the source DNs manually. We can add it in a separate Jira.

          Show
          arpitagarwal Arpit Agarwal added a comment - Also it may be a good idea to add a separate option to source from the most over-utilized DataNodes first so the administrator does not have to pass the source DNs manually. We can add it in a separate Jira.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          +1 for the patch. The test failures look unrelated although couple of the checkstyle issues look valid.

          Show
          arpitagarwal Arpit Agarwal added a comment - +1 for the patch. The test failures look unrelated although couple of the checkstyle issues look valid.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 22m 4s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 9m 49s There were no new javac warning messages.
          +1 javadoc 12m 33s There were no new javadoc warning messages.
          +1 release audit 0m 39s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 3m 37s The applied patch generated 6 new checkstyle issues (total was 99, now 94).
          -1 whitespace 0m 1s The patch has 8 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 45s mvn install still works.
          +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse.
          +1 findbugs 5m 40s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 common tests 24m 54s Tests failed in hadoop-common.
          -1 hdfs tests 176m 3s Tests failed in hadoop-hdfs.
              257m 46s  



          Reason Tests
          Failed unit tests hadoop.net.TestNetUtils
            hadoop.ipc.TestDecayRpcScheduler
            hadoop.ha.TestZKFailoverController
            hadoop.fs.TestLocalFsFCStatistics
            hadoop.ipc.TestIPC
            hadoop.hdfs.TestAppendSnapshotTruncate
            hadoop.hdfs.TestReplaceDatanodeOnFailure
          Timed out tests org.apache.hadoop.cli.TestHDFSCLI



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12750732/h8826_20150816.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 13604bd
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12005/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12005/artifact/patchprocess/whitespace.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/12005/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12005/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12005/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12005/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 22m 4s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 9m 49s There were no new javac warning messages. +1 javadoc 12m 33s There were no new javadoc warning messages. +1 release audit 0m 39s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 3m 37s The applied patch generated 6 new checkstyle issues (total was 99, now 94). -1 whitespace 0m 1s The patch has 8 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 45s mvn install still works. +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse. +1 findbugs 5m 40s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 common tests 24m 54s Tests failed in hadoop-common. -1 hdfs tests 176m 3s Tests failed in hadoop-hdfs.     257m 46s   Reason Tests Failed unit tests hadoop.net.TestNetUtils   hadoop.ipc.TestDecayRpcScheduler   hadoop.ha.TestZKFailoverController   hadoop.fs.TestLocalFsFCStatistics   hadoop.ipc.TestIPC   hadoop.hdfs.TestAppendSnapshotTruncate   hadoop.hdfs.TestReplaceDatanodeOnFailure Timed out tests org.apache.hadoop.cli.TestHDFSCLI Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12750732/h8826_20150816.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 13604bd checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12005/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12005/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/12005/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12005/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12005/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12005/console This message was automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          h8826_20150816.patch: add some tests.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - h8826_20150816.patch: add some tests.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 17m 3s Findbugs (version ) appears to be broken on trunk.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 36s There were no new javac warning messages.
          +1 javadoc 9m 49s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 37s There were no new checkstyle issues.
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 28s mvn install still works.
          +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse.
          +1 findbugs 4m 19s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 common tests 22m 10s Tests failed in hadoop-common.
          -1 hdfs tests 171m 56s Tests failed in hadoop-hdfs.
              236m 54s  



          Reason Tests
          Failed unit tests hadoop.net.TestNetUtils
            hadoop.ha.TestZKFailoverController
          Timed out tests org.apache.hadoop.cli.TestHDFSCLI



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12749979/h8826_20150811.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 7c796fd
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/11971/artifact/patchprocess/whitespace.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/11971/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11971/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11971/testReport/
          Java 1.7.0_55
          uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11971/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 17m 3s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 36s There were no new javac warning messages. +1 javadoc 9m 49s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 37s There were no new checkstyle issues. -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 28s mvn install still works. +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse. +1 findbugs 4m 19s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 common tests 22m 10s Tests failed in hadoop-common. -1 hdfs tests 171m 56s Tests failed in hadoop-hdfs.     236m 54s   Reason Tests Failed unit tests hadoop.net.TestNetUtils   hadoop.ha.TestZKFailoverController Timed out tests org.apache.hadoop.cli.TestHDFSCLI Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12749979/h8826_20150811.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 7c796fd whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/11971/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/11971/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11971/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11971/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11971/console This message was automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          h8826_20150811.patch: adds a new -source option.

          Will add some tests later.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - h8826_20150811.patch: adds a new -source option. Will add some tests later.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          I suggest to add an option to specify the source node list. Then, balancer only selects blocks to move from those nodes.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - I suggest to add an option to specify the source node list. Then, balancer only selects blocks to move from those nodes.

            People

            • Assignee:
              szetszwo Tsz Wo Nicholas Sze
              Reporter:
              szetszwo Tsz Wo Nicholas Sze
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development