Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-16193

Address flaky ExternalAppendOnlyMapSuite spilling tests

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.0.0
    • Fix Version/s: 2.0.0
    • Component/s: Spark Core, Tests
    • Labels:
      None
    • Target Version/s:

      Description

      We've seen tests fail, for different codecs and operations, like so, most recently from 2.0.0 RC1:

      - spilling with compression *** FAILED ***
        java.lang.Exception: Test failed with compression using codec org.apache.spark.io.LZ4CompressionCodec:
      
      assertion failed: expected groupByKey to spill, but did not
        at scala.Predef$.assert(Predef.scala:170)
        at org.apache.spark.TestUtils$.assertSpilled(TestUtils.scala:170)
        at org.apache.spark.util.collection.ExternalAppendOnlyMapSuite.org$apache$spark$util$collection$ExternalAppendOnlyMapSuite$$testSimpleSpilling(ExternalAppendOnlyMapSuite.scala:253)
        at org.apache.spark.util.collection.ExternalAppendOnlyMapSuite$$anonfun$10$$anonfun$apply$mcV$sp$8.apply(ExternalAppendOnlyMapSuite.scala:218)
        at org.apache.spark.util.collection.ExternalAppendOnlyMapSuite$$anonfun$10$$anonfun$apply$mcV$sp$8.apply(ExternalAppendOnlyMapSuite.scala:216)
        at scala.collection.immutable.Stream.foreach(Stream.scala:594)
        ...
      

      My theory is that the listener doesn't receive notification of the spilled stages early enough to show up in the test. It should wait until the job is done before letting the test proceed to query the number of spilled stages.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                srowen Sean R. Owen
                Reporter:
                srowen Sean R. Owen
              • Votes:
                1 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: