[SPARK-17566] "--master yarn --deploy-mode cluster" gives "Launching Python applications through spark-submit is currently only supported for local files"

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.0.0
• Component/s: Spark Submit
• Assignee: Unassigned
• Reporter: Zhenhua Xu
• Created: 16/Sep/16 20:28
• Updated: 18/Sep/16 07:59
• Resolved: 18/Sep/16 03:21

Description

In Spark 1.6, the following command runs fine with both primary and additional python files in hdfs.
/bin/spark-submit --py-files hdfs:///tmp/base.py --master yarn-cluster hdfs:///tmp/pi.py

In Spark 2.0.0, the following command fails:
/bin/spark-submit --py-files hdfs:///tmp/base.py --master yarn --deploy-mode cluster hdfs:///tmp/pi.py

Error:
Launching Python applications through spark-submit is currently only supported for local files: hdfs:///tmp/base.py

Issue Links

• duplicates: SPARK-17512 Specifying remote files for Python based Spark jobs in Yarn cluster mode not working