[SPARK-21792] Document Spark Streaming Dynamic Allocation Configuration

• Type: Documentation
• Status: Resolved
• Priority: Minor
• Resolution: Incomplete
• Affects Version/s: 2.2.0
• Component/s: Documentation, DStreams
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Karthik Palaniappan
• Created: 20/Aug/17 07:06
• Updated: 21/May/19 04:17
• Resolved: 21/May/19 04:17

Description

Support for dynamic allocation was added in https://issues.apache.org/jira/browse/SPARK-12133, and it included new config parameters in https://github.com/apache/spark/blob/branch-2.0/streaming/src/main/scala/org/apache/spark/streaming/scheduler/ExecutorAllocationManager.scala#L189.

These new parameters should be added to http://spark.apache.org/docs/latest/configuration.html.

Out of curiosity: why can't Core's dynamic allocation and Streaming's dynamic allocation both be enabled? It seems like Streaming's dynamic allocation properties should always be used for streaming applications, and Core's should always be ignored. (Either way this should be highlighted in the documentation).