Uploaded image for project: 'TinkerPop'
  1. TinkerPop
  2. TINKERPOP-1754

Spark can not deserialise some ScriptRecordReader parse exceptions

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.1.7
    • Fix Version/s: 3.1.8
    • Component/s: hadoop
    • Labels:
      None

      Description

      ScriptException refer to groovy exception that could point to "Script" class that is not available for system class loader. Spark can not deserialise the exception and user did not get the parse error.
      To fix the problem ScriptRecordReader should not try to propagate all cause exceptions abut only the message with parse error.

      Spark output:

      WARN  [task-result-getter-0] 2017-08-16 11:11:41,777  TaskEndReason.scala:192 - Task exception could not be deserialized
      java.lang.ClassNotFoundException: Script1
      	at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[na:1.8.0_40]
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[na:1.8.0_40]
      	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) ~[na:1.8.0_40]
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[na:1.8.0_40]
      	at java.lang.Class.forName0(Native Method) ~[na:1.8.0_40]
      	at java.lang.Class.forName(Class.java:348) ~[na:1.8.0_40]
      	at org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:67) ~[spark-core_2.11-2.2.0.0-bb4c2a9.jar:2.2.0.0-bb4c2a9]
      	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1613) [na:1.8.0_40]
      	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518) [na:1.8.0_40]
      	at java.io.ObjectInputStream.readClass(ObjectInputStream.java:1484) [na:1.8.0_40]
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1334) [na:1.8.0_40]
      	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) [na:1.8.0_40]
      	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) [na:1.8.0_40]
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) [na:1.8.0_40]
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) [na:1.8.0_40]
      	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) [na:1.8.0_40]
      	at java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:501) [na:1.8.0_40]
      	at java.lang.Throwable.readObject(Throwable.java:914) ~[na:1.8.0_40]
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_40]
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_40]
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_40]
      	at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_40]
      	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) [na:1.8.0_40]
      	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1896) [na:1.8.0_40]
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) [na:1.8.0_40]
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) [na:1.8.0_40]
      	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) [na:1.8.0_40]
      	at java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:501) [na:1.8.0_40]
      	at java.lang.Throwable.readObject(Throwable.java:914) ~[na:1.8.0_40]
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_40]
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_40]
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_40]
      	at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_40]
      	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) [na:1.8.0_40]
      	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1896) [na:1.8.0_40]
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) [na:1.8.0_40]
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) [na:1.8.0_40]
      	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) [na:1.8.0_40]
      	at org.apache.spark.ThrowableSerializationWrapper.readObject(TaskEndReason.scala:193) ~[spark-core_2.11-2.2.0.0-bb4c2a9.jar:2.2.0.0-bb4c2a9]
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_40]
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_40]
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_40]
      	at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_40]
      	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) [na:1.8.0_40]
      	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1896) [na:1.8.0_40]
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) [na:1.8.0_40]
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) [na:1.8.0_40]
      	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) [na:1.8.0_40]
      	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) [na:1.8.0_40]
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) [na:1.8.0_40]
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) [na:1.8.0_40]
      	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) [na:1.8.0_40]
      	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) [na:1.8.0_40]
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) [na:1.8.0_40]
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) [na:1.8.0_40]
      	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) [na:1.8.0_40]
      	at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75) [spark-core_2.11-2.2.0.0-bb4c2a9.jar:2.2.0.0-bb4c2a9]
      	at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114) [spark-core_2.11-2.2.0.0-bb4c2a9.jar:2.2.0.0-bb4c2a9]
      	at org.apache.spark.scheduler.TaskResultGetter$$anon$4$$anonfun$run$2.apply$mcV$sp(TaskResultGetter.scala:136) [spark-core_2.11-2.2.0.0-bb4c2a9.jar:2.2.0.0-bb4c2a9]
      	at org.apache.spark.scheduler.TaskResultGetter$$anon$4$$anonfun$run$2.apply(TaskResultGetter.scala:132) [spark-core_2.11-2.2.0.0-bb4c2a9.jar:2.2.0.0-bb4c2a9]
      	at org.apache.spark.scheduler.TaskResultGetter$$anon$4$$anonfun$run$2.apply(TaskResultGetter.scala:132) [spark-core_2.11-2.2.0.0-bb4c2a9.jar:2.2.0.0-bb4c2a9]
      	at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1954) [spark-core_2.11-2.2.0.0-bb4c2a9.jar:2.2.0.0-bb4c2a9]
      	at org.apache.spark.scheduler.TaskResultGetter$$anon$4.run(TaskResultGetter.scala:132) [spark-core_2.11-2.2.0.0-bb4c2a9.jar:2.2.0.0-bb4c2a9]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_40]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_40]
      	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_40]
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                spmallette Stephen Mallette
                Reporter:
                artem.aliev Artem Aliev
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: