[SPARK-965] Race in DAGSchedulerSuite

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.8.1, 0.9.0
• Fix Version/s: 0.9.0
• Component/s: Spark Core
• Assignee: liancheng
• Reporter: Mark Hamstra
• Created: 25/Nov/13 14:19
• Updated: 07/Dec/13 14:50
• Resolved: 07/Dec/13 14:50

Description

After https://github.com/apache/incubator-spark/pull/159, resubmitFailedStages is scheduled in the eventProcessorActor preStart to run periodically in the actor's thread. But DAGSchedulerSuite calls resubmitFailedStages directly from its own thread, which means that it is possible for both threads to be concurrently mutating shared data structures (e.g. the set of failed stages and cacheLocs), leading to failure of the test calling resubmitFailedStages.