[SPARK-17644] The failed stage never resubmitted due to abort stage in another thread

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.6.0, 2.0.0
• Fix Version/s: 2.0.1, 2.1.0
• Component/s: Scheduler, Spark Core
• Assignee: Fei Wang Assign to me
• Reporter: Fei Wang
• Created: 23/Sep/16 07:43
• Updated: 28/Sep/16 23:30
• Resolved: 28/Sep/16 23:30

Description

there is a race condition when FetchFailed and resubmit failed stage:
job1, job2 run in different threads, if job 1 failed 4 times due to fetchfailed and aborted, then job2 can not post ResubmitFailedStages becase the failedStages in DAGScheduler is not empty now.

Issue Links

• links to: [Github] Pull Request #15213 (scwf)