Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27575

Spark overwrites existing value of spark.yarn.dist.* instead of merging value

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 3.0.0
    • Component/s: Spark Submit, YARN
    • Labels:
      None

      Description

      If we specify `--files` arg when submitting app where configuration has "spark.yarn.dist.files", SparkSubmit overwrites the new files (files provided in arg) instead of merging existing value in configuration.

      Same issue happens also on "spark.yarn.dist.pyFiles", "spark.yarn.dist.jars", "spark.yarn.dist.archives".

       

      While I encountered the issue in Spark 2.4.0, I can see the issue from codebase in master branch and also branch-2.3 as well.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                kabhwan Jungtaek Lim
                Reporter:
                kabhwan Jungtaek Lim
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: