[SPARK-25773] Cancel zombie tasks in a result stage when the job finishes

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 3.0.0
• Component/s: Scheduler, Spark Core
• Assignee: Shixiong Zhu
• Reporter: Shixiong Zhu
• Created: 18/Oct/18 21:06
• Updated: 17/May/20 17:48
• Resolved: 30/Oct/18 17:48

Description

When a job finishes, there may be some zombie tasks still running due to stage retry. Since a result stage will never be used by other jobs, running these tasks are just wasting the cluster resource. This PR just asks TaskScheduler to cancel the running tasks of a result stage when it's already finished. Credits go to @srinathshankar who suggested this idea to me.

Issue Links

• is related to: SPARK-24622 Task attempts in other stage attempts not killed when one task attempt succeeds
• links to: [Github] Pull Request #22771 (zsxwing)
• links to: [Github] Pull Request #22771 (zsxwing)