[SPARK-6869] Add pyspark archives path to PYTHONPATH

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.4.0
• Component/s: PySpark
• Target Version/s: 1.4.0
• Assignee: Lianhui Wang
• Reporter: Weizhong
• Created: 12/Apr/15 08:00
• Updated: 26/Jun/15 08:55
• Resolved: 08/May/15 13:47

Description

From SPARK-1920 and SPARK-1520 we know PySpark on Yarn can not work when the assembly jar are package by JDK 1.7+, so ship pyspark archives to executors by Yarn with --py-files. The pyspark archives name must contains "spark-pyspark".

1st: zip pyspark to spark-pyspark_2.10.zip
2nd:./bin/spark-submit --master yarn-client/yarn-cluster --py-files spark-pyspark_2.10.zip app.py args

Issue Links

• is duplicated by: SPARK-1920 Spark JAR compiled with Java 7 leads to PySpark not working in YARN
• is related to: SPARK-1920 Spark JAR compiled with Java 7 leads to PySpark not working in YARN
• is related to: SPARK-8646 PySpark does not run on YARN if master not provided in command line
• relates to: SPARK-6797 Add support for YARN cluster mode
• relates to: ZEPPELIN-18 Running pyspark without deploying python libraries to every yarn node
• links to: [Github] Pull Request #5478 (Sephiroth-Lin)
• links to: [Github] Pull Request #5580 (lianhuiwang)