Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28214

Flaky test: org.apache.spark.streaming.CheckpointSuite.basic rdd checkpoints + dstream graph checkpoint recovery

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.0.0
    • Component/s: DStreams, Tests
    • Labels:
      None

      Description

      This test has failed a few times in some PRs. Example of a failure:

      Error Message
      org.scalatest.exceptions.TestFailedException: Map() was empty No checkpointed RDDs in state stream before first failure
      Stacktrace
      sbt.ForkMain$ForkError: org.scalatest.exceptions.TestFailedException: Map() was empty No checkpointed RDDs in state stream before first failure
      	at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:528)
      	at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:527)
      	at org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1560)
      	at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:501)
      	at org.apache.spark.streaming.CheckpointSuite.$anonfun$new$3(CheckpointSuite.scala:266)
      	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
      	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
      	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
      	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
      	at org.scalatest.Transformer.apply(Transformer.scala:22)
      	at org.scalatest.Transformer.apply(Transformer.scala:20)
      	at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186)
      	at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:149)
      	at org.scalatest.FunSuiteLike.invokeWithFixture$1(FunSuiteLike.scala:184)
      	at org.scalatest.FunSuiteLike.$anonfun$runTest$1(FunSuiteLike.scala:196)
      	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)
      	at org.scalatest.FunSuiteLike.runTest(FunSuiteLike.scala:196)
      	at org.scalatest.FunSuiteLike.runTest$(FunSuiteLike.scala:178)
      	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:56)
      	at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:221)
      	at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:214)
      	at org.apache.spark.streaming.CheckpointSuite.org$scalatest$BeforeAndAfter$$super$runTest(CheckpointSuite.scala:209)
      

      On top of that, when this failure happens, the test leaves a running SparkContext behind, which makes every single unit test run after it on that project fail.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                kabhwan Jungtaek Lim
                Reporter:
                vanzin Marcelo Masiero Vanzin
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: