[SPARK-28209] Shuffle Storage API: Writer API and usage in BypassMergeSortShuffleWriter

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Shuffle, Spark Core
• Assignee: Matt Cheah
• Reporter: Matt Cheah
• Created: 28/Jun/19 17:55
• Updated: 17/May/20 18:31
• Resolved: 30/Jul/19 21:18

Description

Adds the write-side API for storing shuffle data in arbitrary storage systems. Also refactor the BypassMergeSortShuffleWriter so that it uses this API, demonstrating the API's viability.

Issue Links

• links to: GitHub Pull Request #25007