[SPARK-22466] SPARK_CONF_DIR is not is set by Spark's launch scripts with default value

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.1, 2.2.0
• Fix Version/s: 2.3.0
• Component/s: Spark Submit
• Assignee: Kent Yao 2
• Reporter: Kent Yao 2
• Created: 08/Nov/17 01:56
• Updated: 12/Dec/22 18:10
• Resolved: 09/Nov/17 05:33

Description

If we don't explicitly set `SPARK_CONF_DIR` in `spark-env.sh`, it is only set in `sbin/spark-config.sh` with a default value which is used for spark daemons, but applications with spark-submit will miss this environment variable at runtime.

Issue Links

• causes: SPARK-25934 Mesos: SPARK_CONF_DIR should not be propogated by spark submit
• links to: [Github] Pull Request #19688 (yaooqinn)