[MAPREDUCE-6958] Shuffle audit logger should log size of shuffle transfer

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 2.9.0, 3.0.0-beta1, 2.8.3
• Target Version/s: 2.9.0, 3.0.0-beta1, 2.8.3
• Hadoop Flags: Reviewed
• Assignee: Jason Darrell Lowe
• Reporter: Jason Darrell Lowe
• Created: 14/Sep/17 21:53
• Updated: 19/Sep/17 15:27
• Resolved: 19/Sep/17 15:27

Description

The shuffle audit logger currently logs the job ID and reducer ID but nothing about the size of the requested transfer. It calculates this as part of the HTTP response headers, so it would be trivial to log the response size. This would be very valuable for debugging network traffic storms from the shuffle handler.