[SPARK-27630] Stage retry causes totalRunningTasks calculation to be negative

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 3.0.0
• Component/s: Spark Core
• Assignee: dzcxzl
• Reporter: dzcxzl
• Created: 03/May/19 17:50
• Updated: 08/Jun/20 17:13
• Resolved: 25/Jun/19 19:30

Description

Track tasks separately for each stage attempt (instead of tracking by stage), and do NOT reset the numRunningTasks to 0 on StageCompleted.

In the case of stage retry, the taskEnd event from the zombie stage sometimes makes the number of totalRunningTasks negative, which will causes the job to get stuck.
Similar problem also exists with stageIdToTaskIndices & stageIdToSpeculativeTaskIndices.
If it is a failed taskEnd event of the zombie stage, this will cause stageIdToTaskIndices or stageIdToSpeculativeTaskIndices to remove the task index of the active stage, and the number of totalPendingTasks will increase unexpectedly.

Issue Links

• is related to: SPARK-11334 numRunningTasks can't be less than 0, or it will affect executor allocation
• links to: [Github] Pull Request #24497 (cxzl25)
• links to: [Github] Pull Request #24497 (cxzl25)
• links to: GitHub Pull Request #24497