[SPARK-12133] Support dynamic allocation in Spark Streaming

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: DStreams, Spark Core
• Target Version/s: 2.0.0
• Assignee: Tathagata Das
• Reporter: Andrew Or
• Created: 03/Dec/15 22:22
• Updated: 14/Apr/16 06:09
• Resolved: 14/Apr/16 06:09

Description

Dynamic allocation is a feature that allows your cluster resources to scale up and down based on the workload. Currently it doesn't work well with Spark streaming because of several reasons:

(1) Your executors may never be idle since they run something every N seconds
(2) You should have at least one receiver running always
(3) The existing heuristics don't take into account length of batch queue
...

The goal of this JIRA is to provide better support for using dynamic allocation in streaming. A design doc will be posted shortly.

Issue Links

• relates to: SPARK-10955 Warn if dynamic allocation is enabled for Streaming jobs
• relates to: SPARK-7661 Support for dynamic allocation of resources in Kinesis Spark Streaming
• links to: [Github] Pull Request #12154 (tdas)