[SPARK-46052] Remove unnecessary TaskScheduler.killAllTaskAttempts

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.3, 3.1.3, 3.2.4, 3.3.3, 3.4.1, 3.5.0
• Fix Version/s: 4.0.0
• Component/s: Spark Core
• Labels: pull-request-available
• Assignee: wuyi
• Reporter: wuyi
• Created: 22/Nov/23 08:28
• Updated: 12/Jan/24 09:18
• Resolved: 12/Jan/24 09:18

Description

Spark has two functions to kill all tasks in a Stage:

• `cancelTasks`: Not only kill all the running tasks in all the stage attempts but also abort all the stage attempts
•  `killAllTaskAttempts`: Only kill all the running tasks in all the stage attemtps but won't abort the attempts.

However, there's no use case in Spark that a stage would launch new tasks after its all tasks get killed. So I think we can replace `killAllTaskAttempts` with `cancelTasks` directly.

Issue Links

• links to: GitHub Pull Request #43954