[SPARK-45182] Ignore task completion from old stage after retrying indeterminate stages

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.3.2
• Fix Version/s: 4.0.0, 3.5.1
• Component/s: Spark Core
• Labels: pull-request-available
• Assignee: Mayur Bhosale
• Reporter: Mayur Bhosale
• Created: 15/Sep/23 18:39
• Updated: 26/Sep/23 03:06
• Resolved: 26/Sep/23 03:06

Description

SPARK-25342 Added a support for rolling back shuffle map stage so that all tasks of the stage can be retried when the stage output is indeterminate. This is done by clearing all map outputs at the time of stage submission. This approach workouts well except for this case:

Assume both Shuffle 1 and 2 are indeterminate

ShuffleMapStage1 ––> Shuffle 1 ---–> ShuffleMapStage2 ----> Shuffle 2 ----> ResultStage

• ShuffleMapStage1 is complete
• A task from ShuffleMapStage2 fails with FetchFailed. Other tasks are still running
• Both ShuffleMapStage1 and ShuffleMapStage2 are retried
• ShuffleMapStage1 is retried and completes
• ShuffleMapStage2 reattempt is scheduled for execution
• Before all tasks of ShuffleMapStage2 reattempt could finish, one/more laggard tasks from the original attempt of ShuffleMapStage2 finish and ShuffleMapStage2 also gets marked as complete
• Result Stage gets scheduled and finishes

Internally within Uber, we have been using the stage rollback functionality even for deterministic stages from Spark 2.4.3 to add fault tolerance from server going down in remote shuffle service and have faced this scenario quite often

Ideally, such laggard tasks should not be considered towards the partition completion.

Issue Links

• links to: GitHub Pull Request #42950