[SPARK-37473] BypassMergeSortShuffleWriter may loss data when disk is missing however catagory is present

• Type: Bug
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.4.0, 2.4.1, 2.4.8, 3.0.0, 3.2.0
• Component/s: Shuffle
• Assignee: Unassigned
• Reporter: haiyangyu
• Created: 27/Nov/21 07:07
• Updated: 27/Nov/21 07:14

Description

We think it has no data when the segment file not exists when all segment files produced by `BypassMergeSortShuffleWriter` is merging;

However, `file.exists()` may rerurn `false` when then the disk which segment file in on is missing and the root catagory exists; the missing disk only lead `file.exists()` return `false` but no exception. The task will run in pease and with no current segment file written.

The segment data will be ignored  and leading shuffle data loss.

Issue Links

• links to: [Github] Pull Request #34725 (seayoun)
• links to: [Github] Pull Request #34725 (seayoun)