[CELEBORN-1518] Add support for spark barrier stages

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.6.0, 0.5.2
• Labels: pull-request-available
• Assignee: Mridul Muralidharan
• Reporter: Mridul Muralidharan
• Created: 22/Jul/24 08:31
• Updated: 12/Aug/24 06:25
• Resolved: 12/Aug/24 06:25

Description

Currently, Celeborn does not handle barrier stages in spark.
Barrier stages have an all-or-none property: so any task failure in the stage results in reexecution of all tasks : whether this is due to failure of current stage execution, or due to re-execution of a parent (barrier) stage due to downstream fetch failures.

Other bugs which are related: CELEBORN-1496, CELEBORN-1498

Issue Links

• links to: GitHub Pull Request #2639