[SPARK-31924] Create remote shuffle service reference implementation

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: Shuffle
• Assignee: Unassigned
• Reporter: BoYang
• Shepherd: Felix Cheung
• Created: 07/Jun/20 06:53
• Updated: 04/Oct/20 05:22

Description

People in Spark Scalability & Reliability Sync Meeting have discussed a lot about remote (disaggregated) shuffle service, and plan to do a reference implementation to help demonstrate some basic design and pave the way for a future production grade remote  shuffle service.

 

There are already two pull requests to enhance Spark shuffle metadata API to make it easy/possible to implement remote shuffle service (PR 28616, PR 28618). Creating a remote shuffle service reference implementation will help to validate those shuffle metadata API.

 

Issue Links

• is related to: SPARK-25299 Use remote storage for persisting shuffle data