[SPARK-7009] Build assembly JAR via ant to avoid zip64 problems

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.3.0
• Component/s: Build

Environment

Java 7+

• Assignee: Unassigned
• Reporter: Steve Loughran
• Created: 20/Apr/15 09:53
• Updated: 17/Jun/16 08:33
• Resolved: 02/Feb/16 12:10

Description

SPARK-1911 shows the problem that JDK7+ is using zip64 to build large JARs; a format incompatible with Java and pyspark.

Provided the total number of .class files+resources is <64K, ant can be used to make the final JAR instead, perhaps by unzipping the maven-generated JAR then rezipping it with zip64=never, before publishing the artifact via maven.

Issue Links

• duplicates: SPARK-11157 Allow Spark to be built without assemblies
• relates to: SPARK-1520 Assembly Jar with more than 65536 files won't work when compiled on JDK7 and run on JDK6
• links to: [Github] Pull Request #5592 (steveloughran)
• links to: [Github] Pull Request #5637 (zhzhan)