[SPARK-3796] Create shuffle service for external block storage

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: Spark Core
• Target Version/s: 1.2.0
• Assignee: Aaron Davidson
• Reporter: Patrick Wendell
• Created: 04/Oct/14 19:07
• Updated: 05/Nov/14 23:46
• Resolved: 01/Nov/14 21:38

Description

This task will be broken up into two parts – the first, being to refactor our internal shuffle service to use a BlockTransferService which we can easily extract out into its own service, and then the second is to actually do the extraction.

Here is the design document for the low-level service, nicknamed "Sluice", on top of which will be Spark's BlockTransferService API:
https://docs.google.com/document/d/1zKf3qloBu3dmv2AFyQTwEpumWRPUT5bcAUKB5PGNfx0

Issue Links

• is depended upon by: SPARK-3797 Run the shuffle service inside the YARN NodeManager as an AuxiliaryService
• links to: [Github] Pull Request #3001 (aarondav)