[SPARK-10192] Test for fetch failure in a shared dependency for "skipped" stages

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.6.0
• Component/s: Scheduler, Spark Core, Tests
• Target Version/s: 1.6.0
• Assignee: Imran Rashid
• Reporter: Imran Rashid
• Created: 24/Aug/15 21:17
• Updated: 17/May/20 17:48
• Resolved: 11/Nov/15 00:50

Description

One confusing corner case of the DAGScheduler is when there is a shared shuffle dependency, a job might "skip" the stage associated with that shuffle dependency, since its already been created as part of a different stage. This means if there is a fetch failure, the retry will technically happen as part of a different Stage instance.

This already works, but is lacking tests, so I just plan on adding a simple test case.

Issue Links

• links to: [Github] Pull Request #8402 (squito)
• links to: [Github] Pull Request #9620 (JoshRosen)