[SPARK-30496] Pyspark on kubernetes does not support --py-files from remote storage in cluster mode

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.4
• Fix Version/s: 3.0.0
• Component/s: Kubernetes, PySpark, Spark Core
• Assignee: Unassigned
• Reporter: Navdeep Poonia
• Created: 13/Jan/20 03:48
• Updated: 17/May/20 18:23
• Resolved: 13/Jan/20 11:06

Description

The following spark-submit on yarn works fine downloading the file from remote storage and putting it into PYTHONPATH,

spark-submit --master yarn --deploy-mode cluster --py-files s3://bucket/packages.zip s3://bucket/etl.py

 

While, the same fails on k8s with import errors for packages in the zip file. The following is set to PYTHONPATH on k8s, which has a link to s3 file which can't be supported by PYTHONPATH

PYTHONPATH='/opt/spark/python/lib/pyspark.zip:/opt/spark/python/lib/py4j-*.zip:s3://bucket/packages.zip'

 

Issue Links

• duplicates: SPARK-24736 --py-files not functional for non local URLs. It appears to pass non-local URL's into PYTHONPATH directly.