Details
-
Brainstorming
-
Status: Resolved
-
Major
-
Resolution: Incomplete
-
2.4.0
-
None
Description
The recent PR https://github.com/apache/spark/pull/20669 for removing the init-container introduced a internally used ConfigMap carrying Spark configuration properties in a file for the driver. This ConfigMap gets mounted under $SPARK_HOME/conf and the environment variable SPARK_CONF_DIR is set to point to the mount path. This pretty much prevents users from mounting their own ConfigMaps that carry custom Spark configuration files, e.g., log4j.properties and spark-env.sh and leaves users with only the option of building custom images. IMO, it is very useful to support mounting user-specified ConfigMaps for custom Spark configuration files. This worths further discussions.
Attachments
Issue Links
- Is contained by
-
SPARK-24090 Kubernetes Backend Hotlist for Spark 2.4
- Resolved
- relates to
-
SPARK-24434 Support user-specified driver and executor pod templates
- Resolved