[SPARK-11334] numRunningTasks can't be less than 0, or it will affect executor allocation

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.0
• Fix Version/s: 2.3.0
• Component/s: Spark Core
• Target Version/s: 2.3.0
• Assignee: Sital Kedia
• Reporter: meiyoula
• Created: 27/Oct/15 03:03
• Updated: 06/May/19 17:23
• Resolved: 31/Oct/17 16:51

Description

With Dynamic Allocation function, a task failed over maxFailure time, all the dependent jobs, stages, tasks will be killed or aborted. In this process, SparkListenerTaskEnd event will be behind in SparkListenerStageCompleted and SparkListenerJobEnd. Like the Event Log below:

{"Event":"SparkListenerStageCompleted","Stage Info":{"Stage ID":20,"Stage Attempt ID":0,"Stage Name":"run at AccessController.java:-2","Number of Tasks":200}
{"Event":"SparkListenerJobEnd","Job ID":9,"Completion Time":1444914699829}
{"Event":"SparkListenerTaskEnd","Stage ID":20,"Stage Attempt ID":0,"Task Type":"ResultTask","Task End Reason":{"Reason":"TaskKilled"},"Task Info":{"Task ID":1955,"Index":88,"Attempt":2,"Launch Time":1444914699763,"Executor ID":"5","Host":"linux-223","Locality":"PROCESS_LOCAL","Speculative":false,"Getting Result Time":0,"Finish Time":1444914699864,"Failed":true,"Accumulables":[]}}

Because that, the numRunningTasks in ExecutorAllocationManager class will be less than 0, and it will affect executor allocation.

Issue Links

• is duplicated by: SPARK-18981 The last job hung when speculation is on
• is duplicated by: SPARK-16708 ExecutorAllocationManager.numRunningTasks can be negative when stage retry
• is duplicated by: SPARK-22312 Spark job stuck with no executor due to bug in Executor Allocation Manager
• relates to: SPARK-27630 Stage retry causes totalRunningTasks calculation to be negative
• links to: [Github] Pull Request #9288 (XuTingjun)
• links to: [Github] Pull Request #11205 (jerryshao)
• links to: [Github] Pull Request #19580 (sitalkedia)
• links to: [Github] Pull Request #21166 (sadhen)
• links to: GitHub Pull Request #24497