Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
3.1.0
-
None
Description
SPARK_CONF_DIR hosts configuration files like,
1) spark-defaults.conf - containing all the spark properties.
2) log4j.properties - Logger configuration.
3) spark-env.sh - Environment variables to be setup at driver and executor.
4) core-site.xml - Hadoop related configuration.
5) fairscheduler.xml - Spark's fair scheduling policy at the job level.
6) metrics.properties - Spark metrics.
7) Any user specific - library or framework specific configuration file.
Traditionally, SPARK_CONF_DIR has been the home to all user specific configuration files.
So this feature, will let the user specific configuration files be mounted on the driver and executor pods' SPARK_CONF_DIR.
Please review the attached design doc, for more details.
Attachments
Issue Links
- causes
-
SPARK-34948 Add ownerReference to executor configmap to fix leakages
- Resolved
- links to