Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28535

Flaky test: JobCancellationSuite."interruptible iterator of shuffle reader"

    XMLWordPrintableJSON

    Details

    • Type: Test
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.3.3, 2.4.3, 3.0.0
    • Fix Version/s: 2.3.4, 2.4.4, 3.0.0
    • Component/s: Tests
    • Labels:
      None

      Description

      This is the same flakiness as in SPARK-23881, except the fix there didn't really take, at least on our build machines.

      org.scalatest.exceptions.TestFailedException: 10000 was not less than 10000
            at org.scalatest.Assertions$class.newAssertionFailedException(Assertions.scala:528)
            at org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1560)
            at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:501)
      

      Since that bug is short on explanations, the issue is that there's a race between the thread posting the "stage completed" event to the listener which unblocks the test, and the thread killing the task in the executor. If the even arrives first, it will unblock task execution, and there's a chance that all elements will actually be processed before the executor has a chance to stop the task.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                vanzin Marcelo Masiero Vanzin
                Reporter:
                vanzin Marcelo Masiero Vanzin
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: