[SPARK-13002] Mesos scheduler backend does not follow the property spark.dynamicAllocation.initialExecutors

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5.2, 1.6.0
• Fix Version/s: 2.0.0
• Component/s: Mesos
• Labels: dynamic_allocation mesos
• Target Version/s: 2.0.0
• Assignee: Luc Bourlier
• Reporter: Luc Bourlier
• Created: 26/Jan/16 14:24
• Updated: 05/Feb/16 22:38
• Resolved: 05/Feb/16 22:38

Description

When starting a Spark job on a Mesos cluster, all available cores are reserved (up to spark.cores.max), creating one executor per Mesos node, and as many executors as needed.
This is the case even when dynamic allocation is enabled.

When dynamic allocation is enabled, the number of executor launched at startup should be limited to the value of spark.dynamicAllocation.initialExecutors.

The Mesos scheduler backend already follows the value computed by the ExecutorAllocationManager for the number of executors that should be up and running. Expect at startup, when it just creates all the executors it can.

Issue Links

• relates to: SPARK-13162 Standalone mode does not respect `spark.dynamicAllocation.initialExecutors`
• links to: [Github] Pull Request #11047 (skyluc)