Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3475

Make the replication and invalidation rates configurable

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Trivial Trivial
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.2-alpha
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      This change adds two new configuration parameters.
      # {{dfs.namenode.invalidate.work.pct.per.iteration}} for controlling deletion rate of blocks.
      # {{dfs.namenode.replication.work.multiplier.per.iteration}} for controlling replication rate. This in turn allows controlling the time it takes for decommissioning.

      Please see hdfs-default.xml for detailed description.
      Show
      This change adds two new configuration parameters. # {{dfs.namenode.invalidate.work.pct.per.iteration}} for controlling deletion rate of blocks. # {{dfs.namenode.replication.work.multiplier.per.iteration}} for controlling replication rate. This in turn allows controlling the time it takes for decommissioning. Please see hdfs-default.xml for detailed description.

      Description

      BlockManager currently hardcodes the following two constants:

      private static final int INVALIDATE_WORK_PCT_PER_ITERATION = 32;
      private static final int REPLICATION_WORK_MULTIPLIER_PER_ITERATION = 2;
      

      These are used to throttle/limit the amount of deletion and replication-to-other-DN work done per heartbeat interval of a live DN.

      Not many have had reasons to want these changed so far but there have been a few requests I've faced over the past year from a variety of clusters I've helped maintain. I think with the improvements in disks and network thats already started to be rolled out in production environments out there, changing these may start making sense to some.

      Lets at least make it advanced-configurable with proper docs that warn adequately, with the defaults being what they are today. With hardcodes, it comes down to a recompile for admins, which is not something they may like.

      Please let me know your thoughts.

      1. HDFS-3475.patch
        7 kB
        Harsh J
      2. HDFS-3475.patch
        6 kB
        Harsh J
      3. HDFS-3475.patch
        6 kB
        Harsh J

        Issue Links

          Activity

          Hide
          Aaron T. Myers added a comment -

          I'm +1 to making these configurable.

          Show
          Aaron T. Myers added a comment - I'm +1 to making these configurable.
          Hide
          Harsh J added a comment -

          Hi Aaron - Thanks! Can you take a look at this patch? I believe the existing tests already cover the invalidation and replication work and the defaults are what there were as constants previously. Let me know if you need me to do any more changes!

          Show
          Harsh J added a comment - Hi Aaron - Thanks! Can you take a look at this patch? I believe the existing tests already cover the invalidation and replication work and the defaults are what there were as constants previously. Let me know if you need me to do any more changes!
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12532920/HDFS-3475.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 13 warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2680//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2680//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12532920/HDFS-3475.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 javadoc. The javadoc tool appears to have generated 13 warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2680//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2680//console This message is automatically generated.
          Hide
          Harsh J added a comment -

          The javadocs warnings are not related to this patch. The tests warn I've explained in my previous comment.

          Show
          Harsh J added a comment - The javadocs warnings are not related to this patch. The tests warn I've explained in my previous comment.
          Hide
          Harsh J added a comment -

          Btw I finally played around with the block repl multiplier on a local cluster and I could notice increase in usage when a DN went down and the blocks began moving about, with a higher multiplier value. The logs grepped and counted too showed higher "asking DN to replicate" count per DN heartbeat at NN.

          This should be cautiously increased though, cause even if your network is awesome, your disks cumulatively need to keep up with it anyway. And too much of payload per DN command transfer-back is bad as well. This is why the defaults, I've stuck them as-is and added warnings in documented props.

          Show
          Harsh J added a comment - Btw I finally played around with the block repl multiplier on a local cluster and I could notice increase in usage when a DN went down and the blocks began moving about, with a higher multiplier value. The logs grepped and counted too showed higher "asking DN to replicate" count per DN heartbeat at NN. This should be cautiously increased though, cause even if your network is awesome, your disks cumulatively need to keep up with it anyway. And too much of payload per DN command transfer-back is bad as well. This is why the defaults, I've stuck them as-is and added warnings in documented props.
          Hide
          Harsh J added a comment -

          I noticed I missed out on using one variable (the invalidating pct value). I've fixed that now.

          Show
          Harsh J added a comment - I noticed I missed out on using one variable (the invalidating pct value). I've fixed that now.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12533056/HDFS-3475.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 13 warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery
          org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2685//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2685//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12533056/HDFS-3475.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 javadoc. The javadoc tool appears to have generated 13 warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2685//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2685//console This message is automatically generated.
          Hide
          Aaron T. Myers added a comment -

          One small comment: I think you should add some info to the hdfs-default.xml description for "dfs.namenode.invalidate.work.pct.per.iteration" saying that the value should be between 0-100, or whatever's appropriate. For that matter, since this is a brand new config, you might want to change it to be in the range 0 - 1.0, which I think is a more common way in the Hadoop code base to represent percentages.

          Other than that the patch looks good. +1 pending a fix for the above and an explanation of the two test failures.

          Show
          Aaron T. Myers added a comment - One small comment: I think you should add some info to the hdfs-default.xml description for " dfs.namenode.invalidate.work.pct.per.iteration " saying that the value should be between 0-100, or whatever's appropriate. For that matter, since this is a brand new config, you might want to change it to be in the range 0 - 1.0, which I think is a more common way in the Hadoop code base to represent percentages. Other than that the patch looks good. +1 pending a fix for the above and an explanation of the two test failures.
          Hide
          Harsh J added a comment -

          Updated patch addresses Aaron's comments and also adds in checks for value being -gt zero.

          Both tests mentioned pass for me locally with and without patch applied. Was something on trunk at the time possibly, its gone now.

          [With patch applied]

          -------------------------------------------------------
           T E S T S
          -------------------------------------------------------
          
          -------------------------------------------------------
           T E S T S
          -------------------------------------------------------
          Running org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery
          Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.581 sec
          Running org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
          gTests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.291 sec
          

          Thanks Aaron. After a Jenkins +1 on this new patch, I shall commit it in, unless you have further comments on the patch.

          Show
          Harsh J added a comment - Updated patch addresses Aaron's comments and also adds in checks for value being -gt zero . Both tests mentioned pass for me locally with and without patch applied. Was something on trunk at the time possibly, its gone now. [With patch applied] ------------------------------------------------------- T E S T S ------------------------------------------------------- ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.581 sec Running org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics gTests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.291 sec Thanks Aaron. After a Jenkins +1 on this new patch, I shall commit it in, unless you have further comments on the patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12533831/HDFS-3475.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2717//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2717//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12533831/HDFS-3475.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2717//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2717//console This message is automatically generated.
          Hide
          Eli Collins added a comment -

          +1 Looks good. Style nit going forward: no need to use "this.foo" when referencing members in the constructor, understand you're doing so to be consistent with the surrounding code (which doesn't need to do so either)

          Per offline conversation Harsh tested this on a real cluster with higher multiplier and verfied that re-replication proceeded in kind. Harsh, seems like the current defaults (which you've preserved here) should be revisited? They seem a little low to me, if you concur please file a separate jira for that (to raise them).

          Show
          Eli Collins added a comment - +1 Looks good. Style nit going forward: no need to use "this.foo" when referencing members in the constructor, understand you're doing so to be consistent with the surrounding code (which doesn't need to do so either) Per offline conversation Harsh tested this on a real cluster with higher multiplier and verfied that re-replication proceeded in kind. Harsh, seems like the current defaults (which you've preserved here) should be revisited? They seem a little low to me, if you concur please file a separate jira for that (to raise them).
          Hide
          Eli Collins added a comment -

          I've committed this and merged to branch-2. Thanks Harsh!

          Show
          Eli Collins added a comment - I've committed this and merged to branch-2. Thanks Harsh!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2471 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2471/)
          HDFS-3475. Make the replication monitor multipliers configurable. Contributed by Harsh J Chouraria (Revision 1355089)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1355089
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2471 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2471/ ) HDFS-3475 . Make the replication monitor multipliers configurable. Contributed by Harsh J Chouraria (Revision 1355089) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1355089 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2403 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2403/)
          HDFS-3475. Make the replication monitor multipliers configurable. Contributed by Harsh J Chouraria (Revision 1355089)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1355089
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2403 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2403/ ) HDFS-3475 . Make the replication monitor multipliers configurable. Contributed by Harsh J Chouraria (Revision 1355089) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1355089 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2422 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2422/)
          HDFS-3475. Make the replication monitor multipliers configurable. Contributed by Harsh J Chouraria (Revision 1355089)

          Result = FAILURE
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1355089
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2422 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2422/ ) HDFS-3475 . Make the replication monitor multipliers configurable. Contributed by Harsh J Chouraria (Revision 1355089) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1355089 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1091 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1091/)
          HDFS-3475. Make the replication monitor multipliers configurable. Contributed by Harsh J Chouraria (Revision 1355089)

          Result = FAILURE
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1355089
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1091 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1091/ ) HDFS-3475 . Make the replication monitor multipliers configurable. Contributed by Harsh J Chouraria (Revision 1355089) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1355089 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Hide
          Harsh J added a comment -

          The value for the multiplier we used was 100. This gave us a good boost, but was tried on a cluster that had a very good network gear.

          Yes I do feel the default of 2 is too low today. However, it is low to prevent unnecessary replication in face of long-taking DN restarts and for that reason it does well not to consume too much network. Hence I had left in the defaults. It has to be raised only for fast networks, otherwise the reads/writes may get choked. So I do not think we need to raise it, given the variance of users who use Hadoop.

          Let me know if you feel otherwise Eli.

          P.s. The commit HDFS build failed cause of HDFS-RAID project issues, not HDFS. No alarm, hence.

          Show
          Harsh J added a comment - The value for the multiplier we used was 100. This gave us a good boost, but was tried on a cluster that had a very good network gear. Yes I do feel the default of 2 is too low today. However, it is low to prevent unnecessary replication in face of long-taking DN restarts and for that reason it does well not to consume too much network. Hence I had left in the defaults. It has to be raised only for fast networks, otherwise the reads/writes may get choked. So I do not think we need to raise it, given the variance of users who use Hadoop. Let me know if you feel otherwise Eli. P.s. The commit HDFS build failed cause of HDFS-RAID project issues, not HDFS. No alarm, hence.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1124 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1124/)
          HDFS-3475. Make the replication monitor multipliers configurable. Contributed by Harsh J Chouraria (Revision 1355089)

          Result = FAILURE
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1355089
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1124 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1124/ ) HDFS-3475 . Make the replication monitor multipliers configurable. Contributed by Harsh J Chouraria (Revision 1355089) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1355089 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Hide
          Eli Collins added a comment -

          Thanks Harsh, makes sense.

          Show
          Eli Collins added a comment - Thanks Harsh, makes sense.
          Hide
          Adam Muise added a comment -

          Note on this value: while we tested at 100, this may be too high for a cluster under even moderate workload. The cluster in question had very powerful nodes and dual-bonded 10Gb interfaces. We also had to increase the DataNode memory in the range of 4-6Gb. You may choose to start the value at 10 and go up based on your available memory for the Datanode, I/O, and network capacity.

          Show
          Adam Muise added a comment - Note on this value: while we tested at 100, this may be too high for a cluster under even moderate workload. The cluster in question had very powerful nodes and dual-bonded 10Gb interfaces. We also had to increase the DataNode memory in the range of 4-6Gb. You may choose to start the value at 10 and go up based on your available memory for the Datanode, I/O, and network capacity.
          Hide
          Harsh J added a comment -

          Thanks for following up Adam. Yes, and also since this is a multiplier config, growth of blocks will induce higher memory, so this must only be done if greatly needed and must be carefully monitored.

          Show
          Harsh J added a comment - Thanks for following up Adam. Yes, and also since this is a multiplier config, growth of blocks will induce higher memory, so this must only be done if greatly needed and must be carefully monitored.
          Hide
          Suresh Srinivas added a comment -

          This change adds two new configuration parameters.

          1. dfs.namenode.invalidate.work.pct.per.iteration for controlling deletion rate of blocks.
          2. dfs.namenode.invalidate.work.pct.per.iteration for controlling replication rate. This in turn allows controlling the time it takes for decommissioning.

          Please see hdfs-default.xml for detailed description.

          Show
          Suresh Srinivas added a comment - This change adds two new configuration parameters. dfs.namenode.invalidate.work.pct.per.iteration for controlling deletion rate of blocks. dfs.namenode.invalidate.work.pct.per.iteration for controlling replication rate. This in turn allows controlling the time it takes for decommissioning. Please see hdfs-default.xml for detailed description.

            People

            • Assignee:
              Harsh J
              Reporter:
              Harsh J
            • Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development