When the user defines SPARK_CONF_DIR I think spark should use all the configs available there not only spark-env.
This involves changing SparkSubmitArguments to first read from SPARK_CONF_DIR, and updating the scripts to add SPARK_CONF_DIR to the computed classpath for configs such as log4j, metrics, etc.
I have already prepared a PR for this.
- is duplicated by
-
SPARK-1559 Add conf dir to CLASSPATH in compute-classpath.sh dependent on whether SPARK_CONF_DIR is set
-
- Resolved
-
- relates to
-
SPARK-3620 Refactor config option handling code for spark-submit
-
- Resolved
-
- links to