[SPARK-4286] Support External Shuffle Service with Mesos integration

• Type: Task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.0
• Fix Version/s: 1.4.0
• Component/s: Mesos, Shuffle, Spark Core
• Target Version/s: 1.4.0
• Assignee: Dragos Dascalita Haut
• Reporter: Timothy Chen
• Created: 07/Nov/14 00:49
• Updated: 17/May/20 18:30
• Resolved: 28/Apr/15 19:08

Description

With the new external shuffle service added, we need to also make the Mesos integration able to launch the shuffle service and support the auto scaling executors.

Mesos executor will launch the external shuffle service and leave it running, while have spark executors scalable.

Issue Links

• is duplicated by: SPARK-5197 Support external shuffle service in fine-grained mode on mesos cluster
• links to: [Github] Pull Request #3861 (tnachen)
• links to: [Github] Pull Request #4966 (dragos)
• links to: [Github] Pull Request #4990 (dragos)