[SPARK-40065] Executor ConfigMap is not mounted if profile is not default

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.2.0, 3.2.1, 3.3.0, 3.2.2
• Fix Version/s: 3.3.1, 3.2.3
• Component/s: Kubernetes
• Assignee: Nobuaki Sukegawa
• Reporter: Nobuaki Sukegawa
• Created: 12/Aug/22 22:12
• Updated: 19/Aug/22 19:45
• Resolved: 19/Aug/22 19:44

Description

When executor config map is made optional in SPARK-34316, mount volume is unconditionally disabled erroneously when non-default profile is used.

When spark.kubernetes.executor.disableConfigMap is false, expected behavior is that the ConfigMap is mounted regardless of executor's resource profile. However, it is not mounted if the resource profile is non-default.

Issue Links

• is caused by: SPARK-34316 Support spark.kubernetes.executor.disableConfigMap
• links to: [Github] Pull Request #37504 (nsuke)
• links to: [Github] Pull Request #37504 (nsuke)