Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14133

Add limit of replication streams on a given node at one time for erasurecode blocks resconstruction

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.3.0
    • Fix Version/s: None
    • Component/s: erasure-coding
    • Labels:
      None

      Description

          When i testing blocks reconstruction triger by data disk offline, i found that the Replicated blocks reconstruction speed limit by dfs.namenode.replication.work.multiplier.per.iteration

      in namenode compute work, and for a given node the maximum number of outgoing replication streams at one time limit by used dfs.namenode.replication.max-streams and dfs.namenode.replication.max-streams-hard-limit, but for Erasure Coded blocks only limit total blocks to reconstruct one time by dfs.namenode.replication.work.multiplier.per.iteration

      but not limit maximum of outgoing replication streams per node, so that maybe cause node load imbalance, i think also ec need add node limit.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              liaoyuxiangqin liaoyuxiangqin
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated: