Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
2.3.4, 2.4.0
-
None
Description
If we specify `--files` arg when submitting app where configuration has "spark.yarn.dist.files", SparkSubmit overwrites the new files (files provided in arg) instead of merging existing value in configuration.
Same issue happens also on "spark.yarn.dist.pyFiles", "spark.yarn.dist.jars", "spark.yarn.dist.archives".
While I encountered the issue in Spark 2.4.0, I can see the issue from codebase in master branch and also branch-2.3 as well.