[SPARK-5349] Spark standalone should support dynamic resource scaling

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.2.0
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Tobias Bertelsen
• Created: 21/Jan/15 15:12
• Updated: 20/Jul/15 08:13
• Resolved: 20/Jul/15 08:13

Description

The resource requirements of an interactive shell varies heavily. Sometimes heavy commands are executed, and sometimes the user is thinking, getting coffee, interrupted etc...

A spark shell allocates a fixed number of worker cores (at least in standalone mode). A user thus has the choice to either block other users from the cluster by allocating all cores (default behavior), or restrict him/herself to only a few cores using the option --total-executor-cores. Either way the cores allocated to the shell has low utilization, since they will be waiting for the user a lot.

Instead the spark shell allocate resources directly required to run the driver, and request worker cores only when computation is performed on the RDDs.

This should allow for multiple users, to use an interactive shell concurrently while stille utilizing the entire cluster, when performing heavy operations.

Issue Links

• duplicates: SPARK-4751 Support dynamic allocation for standalone mode
• relates to: SPARK-4922 Support dynamic allocation for coarse-grained Mesos
• relates to: SPARK-3174 Provide elastic scaling within a Spark application