[SPARK-43657] reuse SPARK_CONF_DIR config maps between driver and executor

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.2.4, 3.3.2, 3.4.0
• Component/s: Kubernetes
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: YE
• Created: 22/May/23 08:38
• Updated: 07/Oct/23 00:17

Description

Currently, Spark on k8s-cluster creates two config maps per application: one for the driver and another for the executor. However the config map for executor is almost identical for config map for driver, there's no need to create there two duplicate config maps. As ConfigMaps are object on K8S, there would be some limitations for ConfigMaps on K8S:

1. more config maps means more objects on etcd, and adds overhead to API server
2. Spark driver pod might be ran under limited permission, which means, it might not be possible to create resources rather than exec pod. Therefore driver might not be allowed to create config maps.

I would submit a pr to reuse SPARK_CONF_DIR config maps for running spark on k8s-cluster mode.

Issue Links

• links to: [Github] Pull Request #41257 (advancedxy)
• links to: [Github] Pull Request #41257 (advancedxy)
• links to: GitHub Pull Request #41257