[HDFS-12482] Provide a configuration to adjust the weight of EC recovery tasks to adjust the speed of recovery

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha4
• Fix Version/s: 3.0.0
• Component/s: erasure-coding
• Labels: hdfs-ec-3.0-nice-to-have
• Target Version/s: 3.0.0
• Assignee: Lei (Eddy) Xu
• Reporter: Lei (Eddy) Xu
• Created: 18/Sep/17 20:53
• Updated: 11/Mar/19 11:40
• Resolved: 01/Nov/17 05:08

Description

The relative speed of EC recovery comparing to 3x replica recovery is a function of (EC codec, number of sources, NIC speed, and CPU speed, and etc).

Currently the EC recovery has a fixed xmitsInProgress of max(# of sources, # of targets) comparing to 1 for 3x replica recovery, and NN uses xmitsInProgress to decide how much recovery tasks to schedule to the DataNode this we can add a coefficient for user to tune the weight of EC recovery tasks.

Issue Links

• is duplicated by: HDFS-12483 Provide a configuration to adjust the weight of EC recovery tasks to adjust the speed of recovery