[SPARK-12027] Spark on YARN won't ever ask for more executors than there were containers at time of context creation

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 1.5.2
• Component/s: Scheduler, Spark Core, YARN
• Labels: scheduler scheduling yarn
• Assignee: Unassigned
• Reporter: Hunter Kelly
• Created: 27/Nov/15 12:36
• Updated: 17/May/20 17:47
• Resolved: 30/Nov/15 13:37

Description

Looking at YarnSchedulerBackend, it appears that totalExpectedExecutors is only ever set at startup.

Based on my experience of running on EMR (and a quick browse through the code supports this), Spark will never ask for more executors than what this was set to.

This means that if I add more nodes to my YARN cluster, Spark will never pick them up. This is bad. The whole point of using Spark on EMR/YARN is to be able to add or remove nodes to the cluster and have Spark "Do The Right Thing".