[SPARK-25903] Flaky test: BarrierTaskContextSuite.throw exception on barrier() call timeout

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0, 3.0.0
• Fix Version/s: 2.4.5, 3.0.0
• Component/s: Tests
• Assignee: L. C. Hsieh
• Reporter: Marcelo Masiero Vanzin
• Created: 31/Oct/18 20:08
• Updated: 23/Sep/19 16:16
• Resolved: 23/Sep/19 16:16

Description

We hit this in our internal builds.

Expected exception org.apache.spark.SparkException to be thrown, but no exception was thrown
Stacktrace
      org.scalatest.exceptions.TestFailedException: Expected exception org.apache.spark.SparkException to be thrown, but no exception was thrown
      at org.scalatest.Assertions$class.newAssertionFailedException(Assertions.scala:528)
      at org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1560)
      at org.scalatest.Assertions$class.intercept(Assertions.scala:822)
      at org.scalatest.FunSuite.intercept(FunSuite.scala:1560)
      at org.apache.spark.scheduler.BarrierTaskContextSuite$$anonfun$7.apply(BarrierTaskContextSuite.scala:94)
      at org.apache.spark.scheduler.BarrierTaskContextSuite$$anonfun$7.apply(BarrierTaskContextSuite.scala:76)
      at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)

The problem from the logs is that the first task to call barrier() took a while, and at that point the "slow" task was already running, so the sleep finishes before the 2s timeout runs out.

• https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.7-ubuntu-scala-2.11/169/

Issue Links

• links to: GitHub Pull Request #25897