[SPARK-45498] Followup: Ignore task completion from old stage after retrying indeterminate stages

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 4.0.0, 3.5.1
• Fix Version/s: 4.0.0, 3.5.1
• Component/s: Spark Core
• Labels: pull-request-available
• Assignee: Mayur Bhosale
• Reporter: Mayur Bhosale
• Created: 11/Oct/23 06:30
• Updated: 13/Oct/23 02:19
• Resolved: 13/Oct/23 02:19

Description

With SPARK-45182, we added a fix for not letting laggard tasks of the older attempts of the indeterminate stage from marking the partition has completed in map output tracker.

When a task completes, DAG scheduler also notifies all the tasksets of the stage about that partition being completed. Tasksets would not schedule such task if they are not already scheduled. This is not correct for indeterminate stage, since we want to re-run all the tasks on re-attempt

Issue Links

• links to: GitHub Pull Request #43326