[SPARK-33114] Add metadata in MapStatus to support custom shuffle manager

• Type: Improvement
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.1
• Component/s: Shuffle
• Assignee: Unassigned
• Reporter: BoYang
• Shepherd: Felix Cheung
• Created: 11/Oct/20 05:40
• Updated: 07/Mar/21 08:13

Description

Current MapStatus class is tightly bound with local (sort merge) shuffle which uses BlockManagerId to store the shuffle data location. It could not support other custom shuffle manager implementation.

We could add "metadata" to MapStatus and allow different shuffle manager implementation to store information related to them.

Issue Links

• links to: [Github] Pull Request #30004 (boy-uber)
• links to: [Github] Pull Request #31763 (hiboyang)