[SPARK-48392] Load `spark-defaults.conf` when passing configurations to `spark-submit` through `--properties-file`

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.1
• Fix Version/s: 4.0.0
• Component/s: Spark Core
• Labels: pull-request-available
• Assignee: Chao Sun
• Reporter: Chao Sun
• Created: 22/May/24 20:50
• Updated: 11/Jun/24 14:47
• Resolved: 29/May/24 04:55

Description

Currently, when user pass configurations to spark-submit.sh via --properties-file, the spark-defaults.conf will be completely ignored. This poses issues for some people, for instance, those using Spark on K8S operator from kubeflow. See related issues:

• https://github.com/kubeflow/spark-operator/issues/1183
• https://github.com/kubeflow/spark-operator/issues/1321

Issue Links

• links to: GitHub Pull Request #46709
• links to: GitHub Pull Request #46782