Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-6595

Configure the maximum threads allowed for balancing on datanodes

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.5.0
    • Component/s: balancer & mover, datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently datanode allows a max of 5 threads to be used for balancing.
      In some cases, , it may make sense to use a different number of threads to the purpose of moving.

      1. HDFS-6595.patch
        3 kB
        Benoy Antony
      2. HDFS-6595.patch
        10 kB
        Benoy Antony

        Activity

        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #1813 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1813/)
        HDFS-6595. Allow the maximum threads for balancing on datanodes to be configurable. Contributed by Benoy Antony (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1605565)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1813 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1813/ ) HDFS-6595 . Allow the maximum threads for balancing on datanodes to be configurable. Contributed by Benoy Antony (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1605565 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk #1786 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1786/)
        HDFS-6595. Allow the maximum threads for balancing on datanodes to be configurable. Contributed by Benoy Antony (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1605565)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1786 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1786/ ) HDFS-6595 . Allow the maximum threads for balancing on datanodes to be configurable. Contributed by Benoy Antony (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1605565 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #595 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/595/)
        HDFS-6595. Allow the maximum threads for balancing on datanodes to be configurable. Contributed by Benoy Antony (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1605565)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #595 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/595/ ) HDFS-6595 . Allow the maximum threads for balancing on datanodes to be configurable. Contributed by Benoy Antony (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1605565 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #5779 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5779/)
        HDFS-6595. Allow the maximum threads for balancing on datanodes to be configurable. Contributed by Benoy Antony (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1605565)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #5779 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5779/ ) HDFS-6595 . Allow the maximum threads for balancing on datanodes to be configurable. Contributed by Benoy Antony (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1605565 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
        Hide
        Tsz Wo Nicholas Sze added a comment -

        I have committed this. Thanks, Benoy!

        Show
        Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Benoy!
        Hide
        Tsz Wo Nicholas Sze added a comment -

        +1 patch looks good.

        Show
        Tsz Wo Nicholas Sze added a comment - +1 patch looks good.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12652334/HDFS-6595.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7228//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7228//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12652334/HDFS-6595.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7228//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7228//console This message is automatically generated.
        Hide
        Benoy Antony added a comment -

        Attaching the patch which addresses the comments by Nicholas.
        Also added unit tests to test whether the configuration is effective.

        Show
        Benoy Antony added a comment - Attaching the patch which addresses the comments by Nicholas. Also added unit tests to test whether the configuration is effective.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Benoy, thanks for working on this. Some comments on the patch:

        • Balancer.MAX_NUM_CONCURRENT_MOVES should be moved to DFSConfigKeys, i.e. DFSConfigKeys.DFS_DATANODE_BALANCE_MAX_NUM_CONCURRENT_MOVES_DEFAULT.
        • We should also change Balancer to read the new conf and use it to replace MAX_NUM_CONCURRENT_MOVES.
        Show
        Tsz Wo Nicholas Sze added a comment - Benoy, thanks for working on this. Some comments on the patch: Balancer.MAX_NUM_CONCURRENT_MOVES should be moved to DFSConfigKeys, i.e. DFSConfigKeys.DFS_DATANODE_BALANCE_MAX_NUM_CONCURRENT_MOVES_DEFAULT. We should also change Balancer to read the new conf and use it to replace MAX_NUM_CONCURRENT_MOVES.
        Hide
        Benoy Antony added a comment -

        Thanks for the suggestion, Jitendra Nath Pandey. I'll try that approach.

        Show
        Benoy Antony added a comment - Thanks for the suggestion, Jitendra Nath Pandey . I'll try that approach.
        Hide
        Jitendra Nath Pandey added a comment -

        Its a good idea to add a unit test. I am not clear why we need Balancer to allow more than 5 concurrent moves to test this. Just set the max number of threads to a smaller number (e.g. 3) and see the exception?

        Show
        Jitendra Nath Pandey added a comment - Its a good idea to add a unit test. I am not clear why we need Balancer to allow more than 5 concurrent moves to test this. Just set the max number of threads to a smaller number (e.g. 3) and see the exception?
        Hide
        Benoy Antony added a comment -

        Thanks @jnp. I have tested this manually by increasing the number of concurrent moves on the balancer side. I can add a unit test to the same, but I would need to change the Balancer code also so that it allows to more than 5 concurrent moves per datanode.

        Show
        Benoy Antony added a comment - Thanks @jnp. I have tested this manually by increasing the number of concurrent moves on the balancer side. I can add a unit test to the same, but I would need to change the Balancer code also so that it allows to more than 5 concurrent moves per datanode.
        Hide
        Jitendra Nath Pandey added a comment -

        The patch looks good to me. Since there are no unit tests included, please mention the testing done on the patch.

        Show
        Jitendra Nath Pandey added a comment - The patch looks good to me. Since there are no unit tests included, please mention the testing done on the patch.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12652111/HDFS-6595.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7220//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7220//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12652111/HDFS-6595.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7220//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7220//console This message is automatically generated.
        Hide
        Benoy Antony added a comment -

        Attaching the patch which adds new configuration - dfs.datanode.balance.max.concurrent.moves . The number of threads is set based on this parameter

        Show
        Benoy Antony added a comment - Attaching the patch which adds new configuration - dfs.datanode.balance.max.concurrent.moves . The number of threads is set based on this parameter

          People

          • Assignee:
            Benoy Antony
            Reporter:
            Benoy Antony
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development