[SPARK-3008] PySpark fails due to zipimport not able to load the assembly jar (/usr/bin/python: No module named pyspark)

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Component/s: PySpark
• Labels: pyspark

Environment

Assemebly Jar target/scala-2.10/spark-assembly-1.1.0-SNAPSHOT-hadoop2.2.0.jar

jar -tf assembly/target/scala-2.10/spark-assembly-1.1.0-SNAPSHOT-hadoop2.2.0.jar | wc -l
70441

git sha commit ba28a8fcbc3ba432e7ea4d6f0b535450a6ec96c6

• Target Version/s: 1.1.0
• Assignee: Unassigned
• Reporter: Jai Kumar Singh
• Created: 13/Aug/14 11:52
• Updated: 29/Oct/14 04:02
• Resolved: 18/Aug/14 03:29

Description

PySpark is not working. It fails because zipimport not able to import assembly jar because that contain more than 65536 files.

Email chains in this regard are below

http://mail-archives.apache.org/mod_mbox/incubator-spark-user/201406.mbox/[e-mail]%3E

https://mail.python.org/pipermail/python-list/2014-May/671353.html

Is there any work around to bypass the issue ?

Issue Links

• duplicates: SPARK-1520 Assembly Jar with more than 65536 files won't work when compiled on JDK7 and run on JDK6