[SPARK-43454] SparkConf's get and getAllWithPrefix support variable substitution

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.2.4, 3.3.2, 3.4.0
• Component/s: Kubernetes, Spark Core
• Assignee: Unassigned
• Reporter: YE
• Created: 11/May/23 09:21
• Updated: 15/May/23 04:07

Description

After SPARK-16272:

variable substitution is available for `ConfigEntry`s, this creates some confusion for get conf directly vs get conf from ConfigEntry. For example:

// suppose spark.a=${env:var_a} is set in sparkConf, and it's defined as a ConfigEntry as CONFIG_ENTRY_A

// this would return ${env:var_a}
val a = sparkConf.get("spark.a") 

// this would return the value of var_a in the current JVM process
val configA = sparkConf.get(CONFIG_ENTRY_A)

I'd like to propose that we should support variable substitution for get config directly.

 

I noticed this problem when running spark applications on k8s, and noticed some substitutions are not work as expected. For k8s particularly, it uses getAllWithPrefix extensively, which also suffers this problem

Issue Links

• links to: [Github] Pull Request #41168 (advancedxy)
• links to: [Github] Pull Request #41168 (advancedxy)