Description
If someone sets a spark conf that relates to an existing flag `--master`, we should set it correctly like we do with the defaults file. Otherwise it can have confusing semantics. I noticed this after merging it, otherwise I would have mentioned it in the review.
I think it's as simple as modifying loadDefaults to check the user-supplied options also. We might change it to loadUserProperties since it's no longer just the defaults file.
Attachments
Issue Links
- relates to
-
SPARK-3620 Refactor config option handling code for spark-submit
- Resolved
- links to