[HDFS-11023] I/O based throttling of DN replication work

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0-alpha1
• Assignee: Unassigned
• Reporter: Andrew Wang
• Created: 17/Oct/16 22:19
• Updated: 31/Mar/17 06:23

Description

Datanode recovery work is currently throttled based on the number of blocks to be recovered. This is fine in a replicated world, since each block is equally expensive to recover. However, EC blocks are much more expensive to recover, and the amount depends on the EC policy.

It'd be better to have recovery throttles that accounted for the amount of I/O rather than just the # of blocks.

Issue Links

• relates to: HDFS-9657 Schedule EC tasks at proper time to reduce the impact of recovery traffic