Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25075 Build and test Spark against Scala 2.13
  3. SPARK-32964

Pass all `streaming` module UTs in Scala 2.13

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.1.0
    • 3.1.0
    • DStreams, Spark Core
    • None

    Description

      There is only one failed case of `streaming` module in Scala 2.13:

      • `start with non-serializable DStream checkpoint ` in StreamingContextSuite

      StackOverflowError is thrown here when SerializationDebugger#visit method is called.

      The error stack as follow:

      Expected exception java.io.NotSerializableException to be thrown, but java.lang.StackOverflowError was thrownExpected exception java.io.NotSerializableException to be thrown, but java.lang.StackOverflowError was thrownScalaTestFailureLocation: org.apache.spark.streaming.StreamingContextSuite at (StreamingContextSuite.scala:159)org.scalatest.exceptions.TestFailedException: Expected exception java.io.NotSerializableException to be thrown, but java.lang.StackOverflowError was thrown at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472) at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471) at org.scalatest.funsuite.AnyFunSuite.newAssertionFailedException(AnyFunSuite.scala:1562) at org.scalatest.Assertions.intercept(Assertions.scala:756) at org.scalatest.Assertions.intercept$(Assertions.scala:746) at org.scalatest.funsuite.AnyFunSuite.intercept(AnyFunSuite.scala:1562) at org.apache.spark.streaming.StreamingContextSuite.$anonfun$new$13(StreamingContextSuite.scala:159) ...Caused by: java.lang.StackOverflowError at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1397) at org.apache.spark.streaming.dstream.DStream.writeObject(DStream.scala:513) at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1140) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) at scala.collection.generic.DefaultSerializationProxy.$anonfun$writeObject$1(DefaultSerializationProxy.scala:38) at scala.collection.generic.DefaultSerializationProxy.$anonfun$writeObject$1$adapted(DefaultSerializationProxy.scala:37) at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:553) at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:551) at scala.collection.AbstractIterable.foreach(Iterable.scala:920) at scala.collection.generic.DefaultSerializationProxy.writeObject(DefaultSerializationProxy.scala:37) at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1140) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) at org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visitSerializableWithWriteObjectMethod(SerializationDebugger.scala:230) at org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visitSerializable(SerializationDebugger.scala:189) at org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visit(SerializationDebugger.scala:108) at org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visitSerializableWithWriteObjectMethod(SerializationDebugger.scala:243) at org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visitSerializable(SerializationDebugger.scala:189) at org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visit(SerializationDebugger.scala:108) at org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visitSerializableWithWriteObjectMethod(SerializationDebugger.scala:243) at org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visitSerializable(SerializationDebugger.scala:189) at org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visit(SerializationDebugger.scala:108)
      

      Attachments

        Activity

          People

            LuciferYang Yang Jie
            LuciferYang Yang Jie
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: