[SPARK-4019] Shuffling with more than 2000 reducers may drop all data when partitons are mostly empty or cause deserialization errors if at least one partition is empty

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.2.0
• Fix Version/s: 1.2.0
• Component/s: Spark Core
• Target Version/s: 1.2.0
• Assignee: Josh Rosen
• Reporter: Xiangrui Meng
• Created: 20/Oct/14 22:38
• Updated: 23/Oct/14 23:40
• Resolved: 23/Oct/14 23:40

Description

sc.makeRDD(0 until 10, 1000).repartition(2001).collect()

returns `Array()`.

1.1.0 doesn't have this issue. Tried both HASH and SORT manager.

This problem can also manifest itself as Snappy deserialization errors if the average map output status size is non-zero but there is at least one empty partition, e.g.

sc.makeRDD(0 until 100000, 1000).repartition(2001).collect()

Issue Links

• is related to: SPARK-3740 Use a compressed bitmap to track zero sized blocks in HighlyCompressedMapStatus
• relates to: SPARK-3630 Identify cause of Kryo+Snappy PARSING_ERROR
• links to: [Github] Pull Request #2866 (JoshRosen)