[MAPREDUCE-5873] Shuffle bandwidth computation includes time spent waiting for maps

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.6.0
• Hadoop Flags: Reviewed
• Assignee: Siqi Li Assign to me
• Reporter: Siqi Li
• Created: 01/May/14 20:03
• Updated: 01/Dec/14 03:09
• Resolved: 15/Oct/14 16:04

Description

Currently ShuffleScheduler in ReduceTask JVM status displays bandwidth. Its definition however is confusing because it captures the time where there is no copying because there is a pause between when new wave of map outputs is available.
current bw is definded as (bytes copied so far) / (total time in the copy phase so far)
It would be more useful
1) to measure bandwidth of a single copy call.
2) display aggregated bw as long as there is at least one fetcher is in the copy call.