[SPARK-32658] Partition length number overflow in `PartitionWriterStream`

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.1, 3.1.0
• Component/s: Spark Core
• Target Version/s: 3.0.1, 3.1.0
• Assignee: Xingbo Jiang
• Reporter: Xingbo Jiang
• Created: 19/Aug/20 06:48
• Updated: 20/Aug/20 07:09
• Resolved: 20/Aug/20 07:09

Description

A Spark user reported `FetchFailedException: Stream is corrupted` error when they upgraded their workload to 3.0. The issue happens when the shuffle output data size from a single task is very large (~5GB). The issue is introduced by https://github.com/apache/spark/commit/abef84a868e9e15f346eea315bbab0ec8ac8e389 , the `PartitionWriterStream` defined the partition length to be an int value, while it should be a long value.

Issue Links

• links to: [Github] Pull Request #29474 (jiangxb1987)