[SPARK-24663] Flaky test: StreamingContextSuite "stop slow receiver gracefully"

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.4.4, 3.0.0
• Fix Version/s: 2.4.5, 3.0.0
• Component/s: Tests
• Assignee: Jungtaek Lim
• Reporter: Marcelo Masiero Vanzin
• Created: 27/Jun/18 00:05
• Updated: 14/Sep/19 21:36
• Resolved: 11/Sep/19 20:32

Description

This is another test that sometimes fails on our build machines, although I can't find failures on the riselab jenkins servers. Failure looks like:

org.scalatest.exceptions.TestFailedException: 0 was not greater than 0
      at org.scalatest.Assertions$class.newAssertionFailedException(Assertions.scala:500)
      at org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1555)
      at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:466)
      at org.apache.spark.streaming.StreamingContextSuite$$anonfun$24.apply$mcV$sp(StreamingContextSuite.scala:356)
      at org.apache.spark.streaming.StreamingContextSuite$$anonfun$24.apply(StreamingContextSuite.scala:335)
      at org.apache.spark.streaming.StreamingContextSuite$$anonfun$24.apply(StreamingContextSuite.scala:335)

The test fails in about 2s, while a successful run generally takes 15s. Looking at the logs, the receiver hasn't even started when things fail, which points at a race during test initialization.

Issue Links

• links to: GitHub Pull Request #25725