[SPARK-42170] Files added to the spark-submit command with master K8s and deploy mode cluster, end up in a non deterministic location inside the driver.

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.0, 3.2.2
• Component/s: Kubernetes, Spark Submit
• Assignee: Unassigned
• Reporter: Santosh Pingale
• Created: 24/Jan/23 14:24
• Updated: 24/Jan/23 14:25

Description

Files added to the spark-submit command with master K8s and deploy mode cluster, end up in a non deterministic location inside the driver.

eg:

spark-submit --files myfile --master k8s.. --deploy-mode cluster` will upload the files to /tmp/spark-uuid/myfile

The issue happens because Utils.createTempDir() creates a directory with a uuid in the directory name. This issue does not affect the --archives option, because we `unarchive` the archives into the destination directory which is relative to the working dir. This bug affects file access pre & post app creation. For example if we distribute python dependencies with pex, we need to use --files to attach the pex file and change the spark.pyspark.python to point to this file. But the file location can not be determined before submitting the app. On the other hand, after the app is created, referencing the files without using `SparkFiles.get` also does not work