[SPARK-10842] Eliminate create duplicate stage while generate job dag

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 1.5.0
• Component/s: Scheduler, Spark Core
• Assignee: Unassigned
• Reporter: SuYan
• Created: 26/Sep/15 12:56
• Updated: 12/Dec/22 18:10
• Resolved: 20/Jan/17 11:48

Description

When we traverse RDD, to generate Stage DAG, Spark will skip to judge the stage whether was added into shuffleIdToStage or not in some condition: get shuffleDep from getAncestorShuffleDependency

Issue Links

• duplicates: SPARK-13902 Make DAGScheduler not to create duplicate stage.
• links to: [Github] Pull Request #8923 (suyanNone)