[SPARK-38916] Tasks not killed caused by race conditions between killTask() and launchTask()

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.2.1
• Fix Version/s: 3.3.0
• Component/s: Spark Core
• Assignee: Wei Xue
• Reporter: Wei Xue
• Created: 15/Apr/22 16:33
• Updated: 21/Jul/22 09:08
• Resolved: 21/Apr/22 08:33

Description

Sometimes when the scheduler tries to cancel a task right after it launches that task on the executor, the KillTask and LaunchTask events can come in a reversed order, causing the task to escape the kill-task signal and finish "secretly". And those tasks even show as an un-launched task in Spark UI.

Issue Links

• links to: [Github] Pull Request #36238 (maryannxue)