[SPARK-6287] Add support for dynamic allocation in the Mesos coarse-grained scheduler

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.0
• Fix Version/s: 1.5.0
• Component/s: Mesos
• Target Version/s: 1.5.0
• Assignee: Dragos Dascalita Haut
• Reporter: Dragos Dascalita Haut
• Created: 11/Mar/15 17:37
• Updated: 15/Dec/15 04:03
• Resolved: 09/Jul/15 20:27

Description

Add support inside the coarse-grained Mesos scheduler for dynamic allocation. It amounts to implementing two methods that allow scaling up and down the number of executors:

def doKillExecutors(executorIds: Seq[String])
def doRequestTotalExecutors(requestedTotal: Int)

Issue Links

• duplicates: SPARK-4922 Support dynamic allocation for coarse-grained Mesos
• is related to: SPARK-12330 Mesos coarse executor does not cleanup blockmgr properly on termination if data is stored on disk
• is related to: SPARK-8873 Support cleaning up shuffle files when using shuffle service in Mesos
• links to: [Github] Pull Request #4984 (dragos)