[SPARK-28535] Flaky test: JobCancellationSuite."interruptible iterator of shuffle reader" - ASF JIRA

Attach files

Attach Screenshot

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Test
Status: Resolved
Priority: Minor
Resolution: Fixed
Affects Version/s: 2.3.3, 2.4.3, 3.0.0
Fix Version/s: 2.3.4, 2.4.4, 3.0.0
Component/s: Tests
Labels:
None

Description

This is the same flakiness as in ~~SPARK-23881~~, except the fix there didn't really take, at least on our build machines.

org.scalatest.exceptions.TestFailedException: 10000 was not less than 10000
      at org.scalatest.Assertions$class.newAssertionFailedException(Assertions.scala:528)
      at org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1560)
      at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:501)

Since that bug is short on explanations, the issue is that there's a race between the thread posting the "stage completed" event to the listener which unblocks the test, and the thread killing the task in the executor. If the even arrives first, it will unblock task execution, and there's a chance that all elements will actually be processed before the executor has a chance to stop the task.