Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3814

Make the replication monitor multipliers configurable in 1.x

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.1.0
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      This change adds two new configuration parameters.
      # {{dfs.namenode.invalidate.work.pct.per.iteration}} for controlling deletion rate of blocks.
      # {{dfs.namenode.replication.work.multiplier.per.iteration}} for controlling replication rate. This in turn allows controlling the time it takes for decommissioning.

      Please see hdfs-default.xml for detailed description.
      Show
      This change adds two new configuration parameters. # {{dfs.namenode.invalidate.work.pct.per.iteration}} for controlling deletion rate of blocks. # {{dfs.namenode.replication.work.multiplier.per.iteration}} for controlling replication rate. This in turn allows controlling the time it takes for decommissioning. Please see hdfs-default.xml for detailed description.

      Description

      It would be good to have the equivalent change of HDFS-3475 in 1.x to allow customization of replication rate.

      1. HDFS-3814.patch
        6 kB
        Jing Zhao
      2. HDFS-3814.patch
        11 kB
        Jing Zhao
      3. HDFS-3814.patch
        11 kB
        Jing Zhao
      4. HDFS-3814.patch
        10 kB
        Suresh Srinivas

        Issue Links

          Activity

          Hide
          Suresh Srinivas added a comment -

          We should see if we can squeeze this into 1.1.

          Show
          Suresh Srinivas added a comment - We should see if we can squeeze this into 1.1.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12541427/HDFS-3814.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3038//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541427/HDFS-3814.patch against trunk revision . -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3038//console This message is automatically generated.
          Hide
          Jing Zhao added a comment -

          Added testcases, and change the value of DFS_NAMENODE_INVALIDATE_WORK_PCT_PER_ITERATION_DEFAULT from 32 to 0.32f.

          Show
          Jing Zhao added a comment - Added testcases, and change the value of DFS_NAMENODE_INVALIDATE_WORK_PCT_PER_ITERATION_DEFAULT from 32 to 0.32f.
          Hide
          Suresh Srinivas added a comment -

          Minor comments:

          1. Instead of annotation @exception use @throws
          2. Also use try-invokemethod-fail-catch-and-ignore when testing exceptions. See FileSystemContractBaseTest.java for example.
          Show
          Suresh Srinivas added a comment - Minor comments: Instead of annotation @exception use @throws Also use try-invokemethod-fail-catch-and-ignore when testing exceptions. See FileSystemContractBaseTest.java for example.
          Hide
          Harsh J added a comment -

          Patch seems to deviate a bit from trunk. If we're adding in tests for increase, please do add the same tests to trunk first too? We don't want to be diverging and later regressing.

          Show
          Harsh J added a comment - Patch seems to deviate a bit from trunk. If we're adding in tests for increase, please do add the same tests to trunk first too? We don't want to be diverging and later regressing.
          Hide
          Suresh Srinivas added a comment -

          HDFS-3475 introduced bug will be fixed in HDFS-3816. Thatis the appropriate time to add the tests from this patch.

          Show
          Suresh Srinivas added a comment - HDFS-3475 introduced bug will be fixed in HDFS-3816 . Thatis the appropriate time to add the tests from this patch.
          Hide
          Harsh J added a comment -

          That sounds good to me, thanks Suresh!

          Show
          Harsh J added a comment - That sounds good to me, thanks Suresh!
          Hide
          Suresh Srinivas added a comment -

          Rebased patch.

          Show
          Suresh Srinivas added a comment - Rebased patch.
          Hide
          Suresh Srinivas added a comment -

          I committed the patch to branch-1 and 1.1.0. Thank you Jing.

          Show
          Suresh Srinivas added a comment - I committed the patch to branch-1 and 1.1.0. Thank you Jing.
          Hide
          Ted Yu added a comment -

          The two parameters cited seem to be the same.

          Show
          Ted Yu added a comment - The two parameters cited seem to be the same.
          Hide
          Matt Foley added a comment -

          Closed upon release of Hadoop-1.1.0.

          Show
          Matt Foley added a comment - Closed upon release of Hadoop-1.1.0.

            People

            • Assignee:
              Jing Zhao
              Reporter:
              Suresh Srinivas
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development