Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-4080

"IOException: unexpected exception type" while deserializing tasks

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.1.0, 1.2.0
    • Fix Version/s: 1.1.1, 1.2.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      When deserializing tasks on executors, we sometimes see IOException: unexpected exception type:

       java.io.IOException: unexpected exception type
              java.io.ObjectStreamClass.throwMiscException(ObjectStreamClass.java:1538)
              java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1025)
              java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
              java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
              java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
              java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
              java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
              java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
              java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
              java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
              org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
              org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
              org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:163)
              java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
              java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
              java.lang.Thread.run(Thread.java:745)
      

      Here are some occurrences of this bug reported on the mailing list and GitHub:

      This is probably caused by throwing exceptions other than IOException from our custom readExternal methods (see http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/7u40-b43/java/io/ObjectStreamClass.java#1022). Davies Liu spotted an instance of this in TorrentBroadcast, where a failed require throws a different exception, but this issue has been reported in Spark 1.1.0 as well. To fix this, I'm going to add try-catch blocks around all of our readExternal and writeExternal methods to re-throw caught exceptions as IOException.

      This fix should allow us to determine the actual exceptions that are causing deserialization failures.

        Attachments

          Activity

            People

            • Assignee:
              joshrosen Josh Rosen
              Reporter:
              joshrosen Josh Rosen
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: