[SPARK-6152] Spark does not support Java 8 compiled Scala classes

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.2.1
• Fix Version/s: 1.6.0
• Component/s: Spark Core

Environment

Java 8+
Scala 2.11

• Target Version/s: 1.6.0
• Assignee: Josh Rosen
• Reporter: Ronald Chen
• Created: 04/Mar/15 03:05
• Updated: 13/Nov/15 06:59
• Resolved: 11/Nov/15 19:18

Description

Spark uses reflectasm to check Scala closures which fails if the user defined Scala closures are compiled to Java 8 class version

The cause is reflectasm does not support Java 8
https://github.com/EsotericSoftware/reflectasm/issues/35

Workaround:
Don't compile Scala classes to Java 8, Scala 2.11 does not support nor require any Java 8 features

Stack trace:

java.lang.IllegalArgumentException
	at com.esotericsoftware.reflectasm.shaded.org.objectweb.asm.ClassReader.<init>(Unknown Source)
	at com.esotericsoftware.reflectasm.shaded.org.objectweb.asm.ClassReader.<init>(Unknown Source)
	at com.esotericsoftware.reflectasm.shaded.org.objectweb.asm.ClassReader.<init>(Unknown Source)
	at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$getClassReader(ClosureCleaner.scala:41)
	at org.apache.spark.util.ClosureCleaner$.getInnerClasses(ClosureCleaner.scala:84)
	at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:107)
	at org.apache.spark.SparkContext.clean(SparkContext.scala:1478)
	at org.apache.spark.rdd.RDD.map(RDD.scala:288)
	at ...my Scala 2.11 compiled to Java 8 code calling into spark

Issue Links

• links to: [Github] Pull Request #9512 (JoshRosen)