Uploaded image for project: 'Zeppelin'
  1. Zeppelin
  2. ZEPPELIN-158

Case classes do not persist across nodes/shuffles

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Core, Interpreters
    • Labels:
      None
    • Environment:

      spark-1.2.1, spark-1.3.0

      Description

      Load data from an object file created in spark-shell (1.2.0) using a copy-pasted case class;

      defined class Transaction postETLtraining: org.apache.spark.rdd.RDD[Nothing] = FlatMappedRDD[4957] at objectFile at <console>:177 postETLtest: org.apache.spark.rdd.RDD[Nothing] = FlatMappedRDD[4959] at objectFile at <console>:176 org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 562.0 failed 4 times, most recent failure: Lost task 0.3 in stage 562.0 (TID 13208): java.lang.ClassNotFoundException: $iwC$$iwC$Transaction at org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:65) [etc.]

      Similar errors also occur when attempting to join/orderBy case-class based schemaRDDs. The same code runs fine in a similarly set-up spark-shell (1.2.0 vs 1.2.1, but Zeppelin doesn't build against 1.2.0 due to missing REPL APIs)

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                RPCMoritz Rick Moritz
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: