[HDFS-14133] Add limit of replication streams on a given node at one time for erasurecode blocks resconstruction

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.0
• Component/s: erasure-coding
• Assignee: Unassigned
• Reporter: liaoyuxiangqin
• Created: 07/Dec/18 07:25
• Updated: 10/Dec/18 08:32

Description

    When i testing blocks reconstruction triger by data disk offline, i found that the Replicated blocks reconstruction speed limit by dfs.namenode.replication.work.multiplier.per.iteration

in namenode compute work, and for a given node the maximum number of outgoing replication streams at one time limit by used dfs.namenode.replication.max-streams and dfs.namenode.replication.max-streams-hard-limit, but for Erasure Coded blocks only limit total blocks to reconstruct one time by dfs.namenode.replication.work.multiplier.per.iteration

but not limit maximum of outgoing replication streams per node, so that maybe cause node load imbalance, i think also ec need add node limit.