[SPARK-30985] Propagate SPARK_CONF_DIR files to driver and exec pods.

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.1.0
• Component/s: Kubernetes
• Assignee: Prashant Sharma
• Reporter: Prashant Sharma
• Created: 28/Feb/20 10:48
• Updated: 03/Apr/21 03:31
• Resolved: 16/Nov/20 15:59

Description

SPARK_CONF_DIR hosts configuration files like,
1) spark-defaults.conf - containing all the spark properties.
2) log4j.properties - Logger configuration.
3) spark-env.sh - Environment variables to be setup at driver and executor.
4) core-site.xml - Hadoop related configuration.
5) fairscheduler.xml - Spark's fair scheduling policy at the job level.
6) metrics.properties - Spark metrics.
7) Any user specific - library or framework specific configuration file.

Traditionally, SPARK_CONF_DIR has been the home to all user specific configuration files.

So this feature, will let the user specific configuration files be mounted on the driver and executor pods' SPARK_CONF_DIR.

Please review the attached design doc, for more details.

 

Google docs link

 

Issue Links

• causes: SPARK-34948 Add ownerReference to executor configmap to fix leakages
• links to: GitHub Pull Request #27735