[SPARK-1703] Warn users if Spark is run on JRE6 but compiled with JDK7

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Component/s: Spark Core
• Assignee: Patrick Wendell
• Reporter: Patrick Wendell
• Created: 03/May/14 06:59
• Updated: 03/Mar/15 14:40
• Resolved: 04/May/14 19:22

Description

Right now since the JRE silently swallows the invalid jar, it will produce really confusing behavior if users hit this. We should check if we are in this situation (either in spark-class or compute-classpath) and fail with an explicit error.

We can do something like:

$JAVA_HOME/bin/jar -tf lib/spark-assembly-1.0.0-SNAPSHOT-hadoop1.0.4.jar org/apache/spark/SparkContext

Which, when a user is running with JRE 6 and a JDK-7-compiled jar will produce:

java.util.zip.ZipException: invalid CEN header (bad signature)
	at java.util.zip.ZipFile.open(Native Method)
	at java.util.zip.ZipFile.<init>(ZipFile.java:132)
	at java.util.zip.ZipFile.<init>(ZipFile.java:93)
	at sun.tools.jar.Main.list(Main.java:997)
	at sun.tools.jar.Main.run(Main.java:242)
	at sun.tools.jar.Main.main(Main.java:1167)

Issue Links

• is related to: SPARK-1520 Assembly Jar with more than 65536 files won't work when compiled on JDK7 and run on JDK6
• is related to: SPARK-1911 Warn users if their assembly jars are not built with Java 6
• is related to: SPARK-1698 Improve spark integration