[SPARK-5698] Dynamic allocation: do not allow user to request a negative delta

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.2.0
• Fix Version/s: 1.2.2, 1.3.0
• Component/s: Spark Core, YARN
• Target Version/s: 1.2.2, 1.3.0
• Assignee: Andrew Or
• Reporter: Andrew Or
• Created: 09/Feb/15 22:40
• Updated: 10/Feb/15 01:34
• Resolved: 10/Feb/15 01:34

Description

If the user calls the following in series:

sc.requestExecutors(-5)
sc.killExecutor(...)

Then we might crash the ApplicationMaster. Why?

Well, if we request a negative number of additional executors, then the YarnAllocator will lower the target number of executors it is trying to achieve by 5. This might shoot the target number past 0. Then, if so, when we try to kill an executor it will fail the assertion that the target number must be >=0.

Issue Links

• links to: [Github] Pull Request #4483 (andrewor14)