[SPARK-22312] Spark job stuck with no executor due to bug in Executor Allocation Manager

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.2.0
• Component/s: Scheduler, Spark Core
• Assignee: Unassigned
• Reporter: Sital Kedia
• Created: 19/Oct/17 07:22
• Updated: 17/May/20 19:47
• Resolved: 23/Oct/17 20:27

Description

We often see the issue of Spark jobs stuck because the Executor Allocation Manager does not ask for any executor even if there are pending tasks in case dynamic allocation is turned on. Looking at the logic in EAM which calculates the running tasks, it can happen that the calculation will be wrong and the number of running tasks can become negative.

Issue Links

• duplicates: SPARK-11334 numRunningTasks can't be less than 0, or it will affect executor allocation
• links to: [Github] Pull Request #19534 (sitalkedia)
• links to: [Github] Pull Request #19580 (sitalkedia)