[SPARK-23637] Yarn might allocate more resource if a same executor is killed multiple times.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.3.1, 2.4.0
• Component/s: Spark Core, YARN
• Assignee: Jin Xing
• Reporter: Jin Xing
• Created: 09/Mar/18 05:48
• Updated: 17/May/20 18:13
• Resolved: 04/Apr/18 22:52

Description

YarnAllocator uses numExecutorsRunning to track the number of running executor. numExecutorsRunning is used to check if there're executors missing and need to allocate more.

 In current code, numExecutorsRunning can be negative when driver asks to kill a same idle executor multiple times.

Issue Links

• links to: [Github] Pull Request #20781 (jinxing64)