[SPARK-32922] Add support for ShuffleBlockFetcherIterator to read from merged shuffle partitions and to fallback to original shuffle blocks if encountering failures

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.2.0
• Component/s: Shuffle, Spark Core
• Assignee: Chandni Singh
• Reporter: Min Shen
• Created: 17/Sep/20 17:51
• Updated: 15/Jul/21 23:14
• Resolved: 29/Jun/21 22:45

Description

With the extended MapOutputTracker, the reducers can now get the task input data from the merged shuffle partitions for more efficient shuffle data fetch. The reducers should also be able to fallback to fetching the original unmarked blocks if it encounters failures when fetching the merged shuffle partitions.

Issue Links

• Dependent: SPARK-35671 Add Support in the ESS to serve merged shuffle block meta and data to executors
• requires: SPARK-33350 Add support to DiskBlockManager to create merge directory and to get the local shuffle merged data
• links to: [Github] Pull Request #30768 (otterc)
• links to: [Github] Pull Request #30768 (otterc)
• links to: [Github] Pull Request #32140 (otterc)
• links to: [Github] Pull Request #32140 (otterc)
• links to: [Github] Pull Request #33378 (otterc)
• links to: [Github] Pull Request #33378 (otterc)