[SPARK-29770] Allow setting spark.app.id when spark-submit for Spark on Kubernetes

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 3.0.0
• Component/s: Kubernetes, Spark Core
• Labels: easyfix
• Assignee: Unassigned
• Reporter: Liu Runzhong
• Created: 06/Nov/19 06:20
• Updated: 17/May/20 18:26
• Resolved: 11/Nov/19 22:13

Description

when the user provides  `spark.app.id`  by `spark-submit`, it's actually doing nothing to change the `spark.app.id`, as `spark.app.id` can only be set by `kubernetesAppId` every time, which makes the users feel confused.

https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala#L196

Knowing that `spark.app.id` would be labeled to Driver/Executor pods and other resources and the strict limitation of the label values, but I think it would be more flexible to users to decide how to generate the `spark.app.id` by themselves. 

 

Issue Links

• links to: GitHub Pull Request #26407