[SPARK-14788] Initial number of executors should honor min number if streaming dynamic allocation is enabled

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.0.0
• Component/s: DStreams
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Saisai Shao
• Created: 21/Apr/16 02:05
• Updated: 21/May/19 04:37
• Resolved: 21/May/19 04:37

Description

Streaming dynamic allocation has lower bound spark.streaming.dynamicAllocation.minExecutors and upper bound spark.streaming.dynamicAllocation.maxExecutors of executor number. But seems it is not used when starting on yarn or other cluster manager, I think we should honor minExecutors as an initial number of executors for starting if streaming dynamic allocation is enabled, like what we did in Spark dynamic allocation.

From my understanding this is an issue should be fixed, but I'm not sure it is a by-design choice, what is your opinion andrewor14 and tdas ?