[SPARK-22864] Flaky test: ExecutorAllocationManagerSuite "cancel pending executors when no longer needed"

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.3.0
• Component/s: Tests
• Assignee: Marcelo Masiero Vanzin
• Reporter: Marcelo Masiero Vanzin
• Created: 21/Dec/17 21:36
• Updated: 29/Dec/17 16:52
• Resolved: 29/Dec/17 16:52

Description

Failed in one PR run:

sbt.ForkMain$ForkError: org.scalatest.exceptions.TestFailedException: 0 did not equal -1
	at org.scalatest.Assertions$class.newAssertionFailedException(Assertions.scala:528)
	at org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1560)
	at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:501)
	at org.apache.spark.ExecutorAllocationManagerSuite$$anonfun$9.apply(ExecutorAllocationManagerSuite.scala:271)
	at org.apache.spark.ExecutorAllocationManagerSuite$$anonfun$9.apply(ExecutorAllocationManagerSuite.scala:247)

Logs have a more interesting exception:

17/12/21 04:26:10.846 pool-1-thread-1-ScalaTest-running-ExecutorAllocationManagerSuite INFO ExecutorAllocationManager: Requesting 2 new executors because tasks are backlogged (new desired total will be 5)
17/12/21 04:26:10.858 spark-listener-group-appStatus ERROR AsyncEventQueue: Listener AppStatusListener threw an exception
java.lang.UnsupportedOperationException: duration() called on unfinished task
	at org.apache.spark.scheduler.TaskInfo.duration(TaskInfo.scala:118)
	at org.apache.spark.status.AppStatusListener$$anonfun$onTaskEnd$1.apply(AppStatusListener.scala:468)
	at org.apache.spark.status.AppStatusListener$$anonfun$onTaskEnd$1.apply(AppStatusListener.scala:435)
	at scala.Option.foreach(Option.scala:257)

Seems like a race since it passed several attempts locally.

Issue Links

• links to: [Github] Pull Request #20050 (vanzin)