[SPARK-3613] Don't record the size of each shuffle block for large jobs

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: Shuffle, Spark Core
• Target Version/s: 1.2.0
• Assignee: Reynold Xin
• Reporter: Reynold Xin
• Created: 20/Sep/14 06:53
• Updated: 25/Dec/14 15:21
• Resolved: 30/Sep/14 05:56

Description

MapStatus saves the size of each block (1 byte per block) for a particular map task. This actually means the shuffle metadata is O(M*R), where M = num maps and R = num reduces.

If M is greater than a certain size, we should probably just send an average size instead of a whole array.

Issue Links

• is duplicated by: SPARK-4909 "Error communicating with MapOutputTracker" when run a big spark job
• is related to: SPARK-3740 Use a compressed bitmap to track zero sized blocks in HighlyCompressedMapStatus
• links to: [Github] Pull Request #2470 (rxin)