[SPARK-1615] Very subtle race condition in SparkListenerSuite

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.9.1
• Fix Version/s: 1.0.0
• Component/s: Spark Core
• Assignee: Andrew Or
• Reporter: Andrew Or
• Created: 24/Apr/14 17:33
• Updated: 05/Nov/14 10:45
• Resolved: 25/Apr/14 03:19

Description

Much of SparkListenerSuite relies on LiveListenerBus's waitUntilEmpty() method. As the name suggests, this waits until the event queue is empty. However, the following race condition could happen:

(1) We dequeue the event
(2) The queue is empty, we return true
(3) The test asserts something assuming that all listeners have finished executing (and fails)
(4) The listeners receive the event

This has been a possible race condition for a long time, but for some reason we've never run into it.