[SPARK-35534] Optional source codes uploading in k8s

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.1
• Component/s: Kubernetes
• Assignee: Unassigned
• Reporter: Michal Luscon
• Created: 26/May/21 11:03
• Updated: 26/May/21 11:51

Description

It seems there is currently a hard requirement on uploading files in k8s mode via spark.kubernetes.file.upload.path config option that actually doesn't make sense if your docker image already contains all of the necessary files.

I would propose a local:// extension to spark.kubernetes.file.upload.path, that will point to the local folder containing files without their actual upload.