[SPARK-1911] Warn users if their assembly jars are not built with Java 6

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 1.2.2, 1.3.0
• Component/s: Documentation
• Assignee: Sean R. Owen
• Reporter: Andrew Or
• Created: 22/May/14 23:57
• Updated: 11/Oct/17 01:20
• Resolved: 04/Mar/15 11:44

Description

The root cause of the problem is detailed in: https://issues.apache.org/jira/browse/SPARK-1520.

In short, an assembly jar built with Java 7+ is not always accessible by Python or other versions of Java (especially Java 6). If the assembly jar is not built on the cluster itself, this problem may manifest itself in strange exceptions that are not trivial to debug. This is an issue especially for PySpark on YARN, which relies on the python files included within the assembly jar.

Currently we warn users only in make-distribution.sh, but most users build the jars directly. At the very least we need to emphasize this in the docs (currently missing entirely). The next step is to add a warning prompt in the mvn scripts whenever Java 7+ is detected.

Issue Links

• is related to: SPARK-1520 Assembly Jar with more than 65536 files won't work when compiled on JDK7 and run on JDK6
• relates to: SPARK-1703 Warn users if Spark is run on JRE6 but compiled with JDK7
• relates to: SPARK-1753 PySpark on YARN does not work on assembly jar built on Red Hat based OS
• links to: [Github] Pull Request #4874 (srowen)
• links to: [Github] Pull Request #4888 (srowen)