[CELEBORN-1496] Unable to differentiate map tasks that have only different stage attempts

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.6.0, 0.5.2, 0.4.3
• Labels: pull-request-available
• Assignee: jiang13021
• Reporter: jiang13021
• Created: 09/Jul/24 12:11
• Updated: 02/Sep/24 05:48
• Resolved: 30/Aug/24 01:41

Description

In spark, there is a stage re-computation, the DAGScheduler explicitly calling unregisterAllMapAndMergeOutput and then resubmitting the stage. At this point, Celeborn receives two map task results with only different stage attempt IDs.
For example: "stage 0 stageAttempt 0 map 10 taskAttempt 0" and "stage 0 stageAttempt 1 map 10 taskAttempt 0"

https://github.com/apache/spark/blob/branch-3.2/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L2043

In Celeborn, map results are only distinguished by shuffleId, mapId, and taskAttemptId, hence making it unable to differentiate cases where only the stageAttemptId differs.

Issue Links

• links to: GitHub Pull Request #2609
• links to: GitHub Pull Request #2717