[SPARK-3174] Provide elastic scaling within a Spark application

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.2
• Fix Version/s: 1.2.0
• Component/s: Spark Core, YARN
• Target Version/s: 1.2.0
• Assignee: Andrew Or
• Reporter: Sandy Ryza
• Created: 21/Aug/14 17:36
• Updated: 20/Sep/18 17:21
• Resolved: 05/Nov/14 23:45

Description

A common complaint with Spark in a multi-tenant environment is that applications have a fixed allocation that doesn't grow and shrink with their resource needs. We're blocked on YARN-1197 for dynamically changing the resources within executors, but we can still allocate and discard whole executors.

It would be useful to have some heuristics that

• Request more executors when many pending tasks are building up
• Discard executors when they are idle

See the latest design doc for more information.

Issue Links

• is depended upon by: SPARK-3145 Hive on Spark umbrella
• is related to: HIVE-7768 Integrate with Spark executor scaling [Spark Branch]
• is related to: SPARK-5349 Spark standalone should support dynamic resource scaling
• is related to: YARN-1197 Support changing resources of an allocated container
• relates to: SPARK-4922 Support dynamic allocation for coarse-grained Mesos
• relates to: SPARK-4403 Elastic allocation(spark.dynamicAllocation.enabled) results in task never being execued.
• relates to: SPARK-4751 Support dynamic allocation for standalone mode
• relates to: SPARK-20624 SPIP: Add better handling for node shutdown

Sub-Tasks

• Add scheduler hooks/heuristics for adding and removing executors: Closed
• Expose a mechanism for SparkContext to ask for / remove Yarn containers: Closed
• Create shuffle service for external block storage: Closed
• Run the shuffle service inside the YARN NodeManager as an AuxiliaryService: Closed
• Adding documentations about dynamic resource allocation: Closed