[SPARK-44050] Unable to Mount ConfigMap in Driver Pod - ConfigMap Creation Issue

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 3.3.1
• Component/s: Kubernetes, Spark Submit
• Labels: pull-request-available
• Assignee: Unassigned Assign to me
• Reporter: Harshwardhan Singh Dodiya
• Created: 14/Jun/23 05:36
• Updated: 08/Sep/24 00:25

Description

Dear Spark community,

I am facing an issue related to mounting a ConfigMap in the driver pod of my Spark application. Upon investigation, I realized that the problem is caused by the ConfigMap not being created successfully.

Problem Description:
When attempting to mount the ConfigMap in the driver pod, I encounter consistent failures and my pod stays in containerCreating state. Upon further investigation, I discovered that the ConfigMap does not exist in the Kubernetes cluster, which results in the driver pod's inability to access the required configuration data.

Additional Information:

I would like to highlight that this issue is not a frequent occurrence. It has been observed randomly, affecting the mounting of the ConfigMap in the driver pod only approximately 5% of the time. This intermittent behavior adds complexity to the troubleshooting process, as it is challenging to reproduce consistently.

Error Message:

when describing driver pod (kubectl describe pod pod_name)  get the below error.

"ConfigMap '<configmap-name>' not found."

To Reproduce:

1. Download spark 3.3.1 from https://spark.apache.org/downloads.html

2. create an image with "bin/docker-image-tool.sh"

3. Submit on spark-client via bash command by passing all the details and configurations.

4. Randomly in some of the driver pod we can observe this issue.

 

 

Issue Links

• links to: GitHub Pull Request #45848
• links to: GitHub Pull Request #45911