[ZEPPELIN-158] Case classes do not persist across nodes/shuffles

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Won't Fix
• Component/s: Core, Interpreters

Environment

spark-1.2.1, spark-1.3.0

• Assignee: Unassigned
• Reporter: Rick Moritz
• Created: 07/Jul/15 14:11
• Updated: 08/Jan/16 13:13
• Resolved: 08/Jan/16 13:13

Description

Load data from an object file created in spark-shell (1.2.0) using a copy-pasted case class;

defined class Transaction postETLtraining: org.apache.spark.rdd.RDD[Nothing] = FlatMappedRDD[4957] at objectFile at <console>:177 postETLtest: org.apache.spark.rdd.RDD[Nothing] = FlatMappedRDD[4959] at objectFile at <console>:176 org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 562.0 failed 4 times, most recent failure: Lost task 0.3 in stage 562.0 (TID 13208): java.lang.ClassNotFoundException: $iwC$$iwC$Transaction at org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:65) [etc.]

Similar errors also occur when attempting to join/orderBy case-class based schemaRDDs. The same code runs fine in a similarly set-up spark-shell (1.2.0 vs 1.2.1, but Zeppelin doesn't build against 1.2.0 due to missing REPL APIs)

Issue Links

• is broken by: SPARK-6299 ClassNotFoundException in standalone mode when running groupByKey with class defined in REPL.