[HDFS-9657] Schedule EC tasks at proper time to reduce the impact of recovery traffic

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: erasure-coding
• Labels: hdfs-ec-3.0-nice-to-have
• Assignee: Unassigned
• Reporter: Li Bo
• Created: 18/Jan/16 09:08
• Updated: 01/Sep/17 00:47

Description

The EC recover tasks consume a lot of network bandwidth and disk I/O. Recovering a corrupt block requires transferring 6 blocks , hence creating a 6X overhead in network bandwidth and disk I/O. When a datanode fails , the recovery of the whole blocks on this datanode may use up the network bandwith. We need to start a recovery task at a proper time in order to give less impact to the system.

Issue Links

• is related to: HDFS-11023 I/O based throttling of DN replication work