Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3814

Make the replication monitor multipliers configurable in 1.x

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.1.0
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      This change adds two new configuration parameters.
      # {{dfs.namenode.invalidate.work.pct.per.iteration}} for controlling deletion rate of blocks.
      # {{dfs.namenode.replication.work.multiplier.per.iteration}} for controlling replication rate. This in turn allows controlling the time it takes for decommissioning.

      Please see hdfs-default.xml for detailed description.
      Show
      This change adds two new configuration parameters. # {{dfs.namenode.invalidate.work.pct.per.iteration}} for controlling deletion rate of blocks. # {{dfs.namenode.replication.work.multiplier.per.iteration}} for controlling replication rate. This in turn allows controlling the time it takes for decommissioning. Please see hdfs-default.xml for detailed description.

      Description

      It would be good to have the equivalent change of HDFS-3475 in 1.x to allow customization of replication rate.

      1. HDFS-3814.patch
        6 kB
        Jing Zhao
      2. HDFS-3814.patch
        11 kB
        Jing Zhao
      3. HDFS-3814.patch
        11 kB
        Jing Zhao
      4. HDFS-3814.patch
        10 kB
        Suresh Srinivas

        Issue Links

          Activity

          Suresh Srinivas created issue -
          Jing Zhao made changes -
          Field Original Value New Value
          Assignee Jing Zhao [ jingzhao ]
          Harsh J made changes -
          Link This issue contains HDFS-3475 [ HDFS-3475 ]
          Hide
          Suresh Srinivas added a comment -

          We should see if we can squeeze this into 1.1.

          Show
          Suresh Srinivas added a comment - We should see if we can squeeze this into 1.1.
          Jing Zhao made changes -
          Attachment HDFS-3814.patch [ 12541427 ]
          Jing Zhao made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 1.1.0 [ 12317959 ]
          Affects Version/s 1.0.0 [ 12318243 ]
          Target Version/s 1.2.0 [ 12321657 ] 1.1.0 [ 12317959 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12541427/HDFS-3814.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3038//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541427/HDFS-3814.patch against trunk revision . -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3038//console This message is automatically generated.
          Jing Zhao made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Jing Zhao added a comment -

          Added testcases, and change the value of DFS_NAMENODE_INVALIDATE_WORK_PCT_PER_ITERATION_DEFAULT from 32 to 0.32f.

          Show
          Jing Zhao added a comment - Added testcases, and change the value of DFS_NAMENODE_INVALIDATE_WORK_PCT_PER_ITERATION_DEFAULT from 32 to 0.32f.
          Jing Zhao made changes -
          Attachment HDFS-3814.patch [ 12541443 ]
          Hide
          Suresh Srinivas added a comment -

          Minor comments:

          1. Instead of annotation @exception use @throws
          2. Also use try-invokemethod-fail-catch-and-ignore when testing exceptions. See FileSystemContractBaseTest.java for example.
          Show
          Suresh Srinivas added a comment - Minor comments: Instead of annotation @exception use @throws Also use try-invokemethod-fail-catch-and-ignore when testing exceptions. See FileSystemContractBaseTest.java for example.
          Jing Zhao made changes -
          Attachment HDFS-3814.patch [ 12541449 ]
          Jing Zhao made changes -
          Link This issue relates to HDFS-3816 [ HDFS-3816 ]
          Hide
          Harsh J added a comment -

          Patch seems to deviate a bit from trunk. If we're adding in tests for increase, please do add the same tests to trunk first too? We don't want to be diverging and later regressing.

          Show
          Harsh J added a comment - Patch seems to deviate a bit from trunk. If we're adding in tests for increase, please do add the same tests to trunk first too? We don't want to be diverging and later regressing.
          Hide
          Suresh Srinivas added a comment -

          HDFS-3475 introduced bug will be fixed in HDFS-3816. Thatis the appropriate time to add the tests from this patch.

          Show
          Suresh Srinivas added a comment - HDFS-3475 introduced bug will be fixed in HDFS-3816 . Thatis the appropriate time to add the tests from this patch.
          Hide
          Harsh J added a comment -

          That sounds good to me, thanks Suresh!

          Show
          Harsh J added a comment - That sounds good to me, thanks Suresh!
          Hide
          Suresh Srinivas added a comment -

          Rebased patch.

          Show
          Suresh Srinivas added a comment - Rebased patch.
          Suresh Srinivas made changes -
          Attachment HDFS-3814.patch [ 12541486 ]
          Hide
          Suresh Srinivas added a comment -

          I committed the patch to branch-1 and 1.1.0. Thank you Jing.

          Show
          Suresh Srinivas added a comment - I committed the patch to branch-1 and 1.1.0. Thank you Jing.
          Suresh Srinivas made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Release Note This change adds two new configuration parameters.
          # {{dfs.namenode.invalidate.work.pct.per.iteration}} for controlling deletion rate of blocks.
          # {{dfs.namenode.invalidate.work.pct.per.iteration}} for controlling replication rate. This in turn allows controlling the time it takes for decommissioning.

          Please see hdfs-default.xml for detailed description.
          Fix Version/s 1.1.0 [ 12317959 ]
          Resolution Fixed [ 1 ]
          Hide
          Ted Yu added a comment -

          The two parameters cited seem to be the same.

          Show
          Ted Yu added a comment - The two parameters cited seem to be the same.
          Harsh J made changes -
          Release Note This change adds two new configuration parameters.
          # {{dfs.namenode.invalidate.work.pct.per.iteration}} for controlling deletion rate of blocks.
          # {{dfs.namenode.invalidate.work.pct.per.iteration}} for controlling replication rate. This in turn allows controlling the time it takes for decommissioning.

          Please see hdfs-default.xml for detailed description.
          This change adds two new configuration parameters.
          # {{dfs.namenode.invalidate.work.pct.per.iteration}} for controlling deletion rate of blocks.
          # {{dfs.namenode.replication.work.multiplier.per.iteration}} for controlling replication rate. This in turn allows controlling the time it takes for decommissioning.

          Please see hdfs-default.xml for detailed description.
          Jing Zhao made changes -
          Link This issue relates to HDFS-3819 [ HDFS-3819 ]
          Hide
          Matt Foley added a comment -

          Closed upon release of Hadoop-1.1.0.

          Show
          Matt Foley added a comment - Closed upon release of Hadoop-1.1.0.
          Matt Foley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          4h 7m 1 Jing Zhao 17/Aug/12 23:12
          Patch Available Patch Available Open Open
          2h 5m 1 Jing Zhao 18/Aug/12 01:17
          Open Open Resolved Resolved
          16h 12m 1 Suresh Srinivas 18/Aug/12 17:30
          Resolved Resolved Closed Closed
          60d 1h 56m 1 Matt Foley 17/Oct/12 19:27

            People

            • Assignee:
              Jing Zhao
              Reporter:
              Suresh Srinivas
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development