[MAPREDUCE-5691] Throttle shuffle's bandwidth utilization

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.2.0
• Assignee: Unassigned
• Reporter: Liyin Liang
• Created: 19/Dec/13 07:16
• Updated: 30/Dec/13 02:31

Description

In our hadoop cluster, a reducer of a big job can utilize all the bandwidth during shuffle phase. Then any task reading data from the machine which running that reducer becomes very very slow.
It's better to move DataTransferThrottler from hadoop-hdfs to hadoop-common. And create a throttler for Shuffle to throttle each Fetcher.