[SPARK-42837] spark-submit - issue when resolving dependencies hosted on a private repository in kubernetes cluster mode

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.3.2
• Component/s: Kubernetes, Spark Submit
• Assignee: Unassigned
• Reporter: lione Herbet
• Created: 17/Mar/23 11:19
• Updated: 20/Mar/23 01:14

Description

When using spark operator, if dependencies are hosted on a private repository with authentication needed (like S3 or OCI) the spark operator submitting the job need to have all the secrets to access all dependencies. If not the spark-submit fails.

On a multi tenant kubernetes cluster where the spark operator and spark jobs execution are on seperate namespaces, it involves duplicating all secrets or it won't work.

It seems that spark-submit need to acces dependencies (with credentials) only to resolveGlobPath (https://github.com/apache/spark/blob/v3.3.2/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L364-L367) . It seems to me (but need to be confirmed by someone more skilled than me on spark internals behavior) that this resolveGlobPath task is also done when the driver is downloading the jars.

Would it be possible to have this resolveGlobPath task skipped when running on a  Kubernetes Cluster in cluster mode ?

For example add a condition like this arround the 364-367 lines :

if (isKubernetesCluster) {
...
} 

We could even, for compatibility reason with old behavior if needed, add also a condition on a spark parameter like this :

if (isKubernetesCluster && sparkConf.getBoolean("spark.kubernetes.resolevGlobPathsInSubmit", true)) { 
...
}

i tested both solution locally and it seems to resolve the case.

Do yout think I need to consider other elements ?

I may submit a patch depending on your feedback