[SPARK-3740] Use a compressed bitmap to track zero sized blocks in HighlyCompressedMapStatus

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: Shuffle, Spark Core
• Labels: starter
• Target Version/s: 1.2.0
• Assignee: Josh Rosen
• Reporter: Reynold Xin
• Created: 30/Sep/14 05:58
• Updated: 20/Nov/14 07:14
• Resolved: 20/Nov/14 07:14

Description

HighlyCompressedMapStatus uses a single long to track the average block size. However, if a stage has a lot of zero sized outputs, this leads to inefficiency because executors would need to send requests to fetch zero sized blocks.

We can use a compressed bitmap to track the zero-sized blocks.

See discussion in https://github.com/apache/spark/pull/2470

Issue Links

• relates to: SPARK-4019 Shuffling with more than 2000 reducers may drop all data when partitons are mostly empty or cause deserialization errors if at least one partition is empty
• relates to: SPARK-3613 Don't record the size of each shuffle block for large jobs
• links to: [Github] Pull Request #2866 (JoshRosen)