[SPARK-37023] Avoid fetching merge status when shuffleMergeEnabled is false for a shuffleDependency during retry

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.1, 3.3.0
• Component/s: Shuffle
• Assignee: Minchu Yang
• Reporter: Ye Zhou
• Created: 16/Oct/21 06:44
• Updated: 11/Nov/21 09:03
• Resolved: 11/Nov/21 09:03

Description

The assertion below in MapOutoutputTracker.getMapSizesByExecutorId is not guaranteed

assert(mapSizesByExecutorId.enableBatchFetch == true)

The reason is during some stage retry cases, the shuffleDependency.shuffleMergeEnabled is set to false, but there will be mergeStatus since the Driver has collected the merged status for its shuffle dependency. If this is the case, the current implementation would set the enableBatchFetch to false, since there are mergeStatus.

Details can be found here:

https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/MapOutputTracker.scala#L1492

We should improve the implementation here.

Issue Links

• links to: [Github] Pull Request #34461 (rmcyang)
• links to: [Github] Pull Request #34461 (rmcyang)