Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-4660

JavaSerializer uses wrong classloader

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.1.0, 1.1.1
    • Fix Version/s: 1.1.2, 1.2.1, 1.3.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      During testing we found failures when trying to load some classes of the user application:

      ERROR 2014-11-29 20:01:56 org.apache.spark.storage.BlockManagerWorker: Exception handling buffer message
      java.lang.ClassNotFoundException: org.apache.spark.demo.HttpReceiverCases$HttpRequest
      	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
      	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
      	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
      	at java.lang.Class.forName0(Native Method)
      	at java.lang.Class.forName(Class.java:270)
      	at org.apache.spark.serializer.JavaDeseriali
      zationStream$$anon$1.resolveClass(JavaSerializer.scala:59)
      	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1612)
      	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
      	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
      	at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
      	at org.apache.spark.serializer.DeserializationStream$$anon$1.getNext(Serializer.scala:133)
      	at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71)
      	at org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:235)
      	at org.apache.spark.storage.MemoryStore.putIterator(MemoryStore.scala:126)
      	at org.apache.spark.storage.MemoryStore.putIterator(MemoryStore.scala:104)
      	at org.apache.spark.storage.MemoryStore.putBytes(MemoryStore.scala:76)
      	at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:748)
      	at org.apache.spark.storage.BlockManager.putBytes(BlockManager.scala:639)
      	at org.apache.spark.storage.BlockManagerWorker.putBlock(BlockManagerWorker.scala:92)
      	at org.apache.spark.storage.BlockManagerWorker.processBlockMessage(BlockManagerWorker.scala:73)
      	at org.apache.spark.storage.BlockManagerWorker$$anonfun$2.apply(BlockManagerWorker.scala:48)
      	at org.apache.spark.storage.BlockManagerWorker$$anonfun$2.apply(BlockManagerWorker.scala:48)
      	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
      	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
      	at scala.collection.Iterator$class.foreach(Iterator.scala:727)
      	at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
      	at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
      	at org.apache.spark.storage.BlockMessageArray.foreach(BlockMessageArray.scala:28)
      	at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
      	at org.apache.spark.storage.BlockMessageArray.map(BlockMessageArray.scala:28)
      	at org.apache.spark.storage.BlockManagerWorker.onBlockMessageReceive(BlockManagerWorker.scala:48)
      	at org.apache.spark.storage.BlockManagerWorker$$anonfun$1.apply(BlockManagerWorker.scala:38)
      	at org.apache.spark.storage.BlockManagerWorker$$anonfun$1.apply(BlockManagerWorker.scala:38)
      	at org.apache.spark.network.ConnectionManager.org$apache$spark$network$ConnectionManager$$handleMessage(ConnectionManager.scala:682)
      	at org.apache.spark.network.ConnectionManager$$anon$10.run(ConnectionManager.scala:520)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:744)
      

        Attachments

        1. spark-serializer-classloader.patch
          0.7 kB
          Piotr Kolaczkowski

          Issue Links

            Activity

              People

              • Assignee:
                pkolaczk Piotr Kolaczkowski
                Reporter:
                pkolaczk Piotr Kolaczkowski
              • Votes:
                1 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: