[SPARK-4751] Support dynamic allocation for standalone mode

• Type: New Feature
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.2.0
• Fix Version/s: 1.5.0
• Component/s: Spark Core
• Target Version/s: 1.5.0
• Assignee: Andrew Or
• Reporter: Andrew Or
• Created: 05/Dec/14 00:37
• Updated: 26/Oct/15 20:19
• Resolved: 01/Aug/15 18:57

Description

This is equivalent to SPARK-3822 but for standalone mode.

This is actually a very tricky issue because the scheduling mechanism in the standalone Master uses different semantics. In standalone mode we allocate resources based on cores. By default, an application will grab all the cores in the cluster unless "spark.cores.max" is specified. Unfortunately, this means an application could get executors of different sizes (in terms of cores) if:

1) App 1 kills an executor
2) App 2, with "spark.cores.max" set, grabs a subset of cores on a worker
3) App 1 requests an executor

In this case, the new executor that App 1 gets back will be smaller than the rest and can execute fewer tasks in parallel. Further, standalone mode is subject to the constraint that only one executor can be allocated on each worker per application. As a result, it is rather meaningless to request new executors if the existing ones are already spread out across all nodes.

Issue Links

• is duplicated by: SPARK-5349 Spark standalone should support dynamic resource scaling
• is related to: SPARK-3174 Provide elastic scaling within a Spark application
• relates to: SPARK-4922 Support dynamic allocation for coarse-grained Mesos
• links to: [Github] Pull Request #7532 (andrewor14)