[SPARK-8017] YARN cluster python --py-files does not work

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.4.0
• Component/s: PySpark, YARN
• Target Version/s: 1.4.1, 1.5.0
• Assignee: Unassigned
• Reporter: Andrew Or
• Created: 01/Jun/15 18:21
• Updated: 01/Jun/15 18:24
• Resolved: 01/Jun/15 18:24

Description

When I ran the following, it works in client mode but not cluster mode

bin/spark-submit --master yarn --deploy-mode X --py-files secondary.py app.py

where app.py depends on secondary.py.

Python YARN cluster mode is added recently so this is not a blocker.

Issue Links

• duplicates: SPARK-5479 PySpark on yarn mode need to support non-local python files