Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3475

Make the replication and invalidation rates configurable

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Trivial Trivial
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.2-alpha
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      This change adds two new configuration parameters.
      # {{dfs.namenode.invalidate.work.pct.per.iteration}} for controlling deletion rate of blocks.
      # {{dfs.namenode.replication.work.multiplier.per.iteration}} for controlling replication rate. This in turn allows controlling the time it takes for decommissioning.

      Please see hdfs-default.xml for detailed description.
      Show
      This change adds two new configuration parameters. # {{dfs.namenode.invalidate.work.pct.per.iteration}} for controlling deletion rate of blocks. # {{dfs.namenode.replication.work.multiplier.per.iteration}} for controlling replication rate. This in turn allows controlling the time it takes for decommissioning. Please see hdfs-default.xml for detailed description.

      Description

      BlockManager currently hardcodes the following two constants:

      private static final int INVALIDATE_WORK_PCT_PER_ITERATION = 32;
      private static final int REPLICATION_WORK_MULTIPLIER_PER_ITERATION = 2;
      

      These are used to throttle/limit the amount of deletion and replication-to-other-DN work done per heartbeat interval of a live DN.

      Not many have had reasons to want these changed so far but there have been a few requests I've faced over the past year from a variety of clusters I've helped maintain. I think with the improvements in disks and network thats already started to be rolled out in production environments out there, changing these may start making sense to some.

      Lets at least make it advanced-configurable with proper docs that warn adequately, with the defaults being what they are today. With hardcodes, it comes down to a recompile for admins, which is not something they may like.

      Please let me know your thoughts.

      1. HDFS-3475.patch
        6 kB
        Harsh J
      2. HDFS-3475.patch
        6 kB
        Harsh J
      3. HDFS-3475.patch
        7 kB
        Harsh J

        Issue Links

          Activity

          Harsh J created issue -
          Harsh J made changes -
          Field Original Value New Value
          Attachment HDFS-3475.patch [ 12532920 ]
          Harsh J made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 2.0.0-alpha [ 12320353 ]
          Affects Version/s 3.0.0 [ 12320356 ]
          Target Version/s 3.0.0 [ 12320356 ] 2.0.1-alpha, 3.0.0 [ 12321440, 12320356 ]
          Harsh J made changes -
          Attachment HDFS-3475.patch [ 12533056 ]
          Harsh J made changes -
          Attachment HDFS-3475.patch [ 12533831 ]
          Eli Collins made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Target Version/s 2.0.1-alpha, 3.0.0 [ 12321440, 12320356 ]
          Fix Version/s 2.0.1-alpha [ 12321440 ]
          Resolution Fixed [ 1 ]
          Harsh J made changes -
          Link This issue Is contained by HDFS-3814 [ HDFS-3814 ]
          Jing Zhao made changes -
          Link This issue relates to HDFS-3816 [ HDFS-3816 ]
          Suresh Srinivas made changes -
          Release Note This change adds two new configuration parameters.
          # {{dfs.namenode.invalidate.work.pct.per.iteration}} for controlling deletion rate of blocks.
          # {{dfs.namenode.replication.work.multiplier.per.iteration}} for controlling replication rate. This in turn allows controlling the time it takes for decommissioning.

          Please see hdfs-default.xml for detailed description.
          Jing Zhao made changes -
          Link This issue relates to HDFS-3819 [ HDFS-3819 ]
          Arun C Murthy made changes -
          Fix Version/s 2.0.2-alpha [ 12322472 ]
          Fix Version/s 2.1.0-alpha [ 12321440 ]
          Eli Collins made changes -
          Summary Make the replication monitor multipliers configurable Make the replication and invalidation multipliers configurable
          Eli Collins made changes -
          Summary Make the replication and invalidation multipliers configurable Make the replication and invalidation rates configurable
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Harsh J
              Reporter:
              Harsh J
            • Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development