Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-6595

Configure the maximum threads allowed for balancing on datanodes

        Details

        • Type: Improvement Improvement
        • Status: Closed
        • Priority: Minor Minor
        • Resolution: Fixed
        • Affects Version/s: None
        • Fix Version/s: 2.5.0
        • Component/s: balancer & mover, datanode
        • Labels:
          None
        • Hadoop Flags:
          Reviewed

          Description

          Currently datanode allows a max of 5 threads to be used for balancing.
          In some cases, , it may make sense to use a different number of threads to the purpose of moving.

          1. Text File
            HDFS-6595.patch
            10 kB
            Benoy Antony
          2. Text File
            HDFS-6595.patch
            3 kB
            Benoy Antony

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            Benoy Antony added a comment -

            Attaching the patch which adds new configuration - dfs.datanode.balance.max.concurrent.moves . The number of threads is set based on this parameter

            Show
            Benoy Antony added a comment - Attaching the patch which adds new configuration - dfs.datanode.balance.max.concurrent.moves . The number of threads is set based on this parameter
            Hide
            Permalink
            Hadoop QA added a comment -

            -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12652111/HDFS-6595.patch
            against trunk revision .

            +1 @author. The patch does not contain any @author tags.

            -1 tests included. The patch doesn't appear to include any new or modified tests.
            Please justify why no new tests are needed for this patch.
            Also please list what manual steps were performed to verify this patch.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 javadoc. There were no new javadoc warning messages.

            +1 eclipse:eclipse. The patch built with eclipse:eclipse.

            +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7220//testReport/
            Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7220//console

            This message is automatically generated.

            Show
            Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12652111/HDFS-6595.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7220//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7220//console This message is automatically generated.
            Hide
            Permalink
            Jitendra Nath Pandey added a comment -

            The patch looks good to me. Since there are no unit tests included, please mention the testing done on the patch.

            Show
            Jitendra Nath Pandey added a comment - The patch looks good to me. Since there are no unit tests included, please mention the testing done on the patch.
            Hide
            Permalink
            Benoy Antony added a comment -

            Thanks @jnp. I have tested this manually by increasing the number of concurrent moves on the balancer side. I can add a unit test to the same, but I would need to change the Balancer code also so that it allows to more than 5 concurrent moves per datanode.

            Show
            Benoy Antony added a comment - Thanks @jnp. I have tested this manually by increasing the number of concurrent moves on the balancer side. I can add a unit test to the same, but I would need to change the Balancer code also so that it allows to more than 5 concurrent moves per datanode.
            Hide
            Permalink
            Jitendra Nath Pandey added a comment -

            Its a good idea to add a unit test. I am not clear why we need Balancer to allow more than 5 concurrent moves to test this. Just set the max number of threads to a smaller number (e.g. 3) and see the exception?

            Show
            Jitendra Nath Pandey added a comment - Its a good idea to add a unit test. I am not clear why we need Balancer to allow more than 5 concurrent moves to test this. Just set the max number of threads to a smaller number (e.g. 3) and see the exception?
            Hide
            Permalink
            Benoy Antony added a comment -

            Thanks for the suggestion, Jitendra Nath Pandey. I'll try that approach.

            Show
            Benoy Antony added a comment - Thanks for the suggestion, Jitendra Nath Pandey . I'll try that approach.
            Hide
            Permalink
            Tsz Wo Nicholas Sze added a comment -

            Benoy, thanks for working on this. Some comments on the patch:

            • Balancer.MAX_NUM_CONCURRENT_MOVES should be moved to DFSConfigKeys, i.e. DFSConfigKeys.DFS_DATANODE_BALANCE_MAX_NUM_CONCURRENT_MOVES_DEFAULT.
            • We should also change Balancer to read the new conf and use it to replace MAX_NUM_CONCURRENT_MOVES.
            Show
            Tsz Wo Nicholas Sze added a comment - Benoy, thanks for working on this. Some comments on the patch: Balancer.MAX_NUM_CONCURRENT_MOVES should be moved to DFSConfigKeys, i.e. DFSConfigKeys.DFS_DATANODE_BALANCE_MAX_NUM_CONCURRENT_MOVES_DEFAULT. We should also change Balancer to read the new conf and use it to replace MAX_NUM_CONCURRENT_MOVES.
            Hide
            Permalink
            Benoy Antony added a comment -

            Attaching the patch which addresses the comments by Nicholas.
            Also added unit tests to test whether the configuration is effective.

            Show
            Benoy Antony added a comment - Attaching the patch which addresses the comments by Nicholas. Also added unit tests to test whether the configuration is effective.
            Hide
            Permalink
            Hadoop QA added a comment -

            +1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12652334/HDFS-6595.patch
            against trunk revision .

            +1 @author. The patch does not contain any @author tags.

            +1 tests included. The patch appears to include 1 new or modified test files.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 javadoc. There were no new javadoc warning messages.

            +1 eclipse:eclipse. The patch built with eclipse:eclipse.

            +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7228//testReport/
            Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7228//console

            This message is automatically generated.

            Show
            Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12652334/HDFS-6595.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7228//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7228//console This message is automatically generated.
            Hide
            Permalink
            Tsz Wo Nicholas Sze added a comment -

            +1 patch looks good.

            Show
            Tsz Wo Nicholas Sze added a comment - +1 patch looks good.
            Hide
            Permalink
            Tsz Wo Nicholas Sze added a comment -

            I have committed this. Thanks, Benoy!

            Show
            Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Benoy!
            Hide
            Permalink
            Hudson added a comment -

            SUCCESS: Integrated in Hadoop-trunk-Commit #5779 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5779/)
            HDFS-6595. Allow the maximum threads for balancing on datanodes to be configurable. Contributed by Benoy Antony (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1605565)

            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
            Show
            Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #5779 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5779/ ) HDFS-6595 . Allow the maximum threads for balancing on datanodes to be configurable. Contributed by Benoy Antony (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1605565 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
            Hide
            Permalink
            Hudson added a comment -

            FAILURE: Integrated in Hadoop-Yarn-trunk #595 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/595/)
            HDFS-6595. Allow the maximum threads for balancing on datanodes to be configurable. Contributed by Benoy Antony (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1605565)

            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
            Show
            Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #595 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/595/ ) HDFS-6595 . Allow the maximum threads for balancing on datanodes to be configurable. Contributed by Benoy Antony (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1605565 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
            Hide
            Permalink
            Hudson added a comment -

            SUCCESS: Integrated in Hadoop-Hdfs-trunk #1786 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1786/)
            HDFS-6595. Allow the maximum threads for balancing on datanodes to be configurable. Contributed by Benoy Antony (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1605565)

            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
            Show
            Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1786 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1786/ ) HDFS-6595 . Allow the maximum threads for balancing on datanodes to be configurable. Contributed by Benoy Antony (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1605565 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
            Hide
            Permalink
            Hudson added a comment -

            FAILURE: Integrated in Hadoop-Mapreduce-trunk #1813 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1813/)
            HDFS-6595. Allow the maximum threads for balancing on datanodes to be configurable. Contributed by Benoy Antony (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1605565)

            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java
            • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
            Show
            Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1813 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1813/ ) HDFS-6595 . Allow the maximum threads for balancing on datanodes to be configurable. Contributed by Benoy Antony (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1605565 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java

              People

              • Assignee:
                Benoy Antony
                Reporter:
                Benoy Antony
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Development