[SPARK-5479] PySpark on yarn mode need to support non-local python files

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.0
• Fix Version/s: 1.5.0
• Component/s: PySpark, YARN
• Target Version/s: 1.5.0
• Assignee: Marcelo Masiero Vanzin
• Reporter: Lianhui Wang
• Created: 29/Jan/15 09:53
• Updated: 03/Aug/15 20:24
• Resolved: 10/Jun/15 20:28

Description

In SPARK-5162 vgrigor reports this:
Now following code cannot work:
aws emr add-steps --cluster-id "j-XYWIXMD234" \
--steps Name=SparkPi,Jar=s3://eu-west-1.elasticmapreduce/libs/script-runner/script-runner.jar,Args=[/home/hadoop/spark/bin/spark-submit,--deploy-mode,cluster,--master,yarn-cluster,--py-files,s3://mybucketat.amazonaws.com/tasks/main.py,main.py,param1],ActionOnFailure=CONTINUE

so we need to support non-local python files on yarn client and cluster mode.
before submitting application to Yarn, we need to download non-local files to local or hdfs path.
or spark.yarn.dist.files need to support other non-local files.

Issue Links

• is duplicated by: SPARK-7725 --py-files doesn't seem to work in YARN cluster mode
• is duplicated by: SPARK-8017 YARN cluster python --py-files does not work
• relates to: SPARK-5162 Python yarn-cluster mode
• links to: [Github] Pull Request #6360 (vanzin)