Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-10436

spark-submit overwrites spark.files defaults with the job script filename

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Duplicate
    • Affects Version/s: 1.4.0
    • Fix Version/s: None
    • Component/s: Spark Submit
    • Labels:
    • Environment:

      Ubuntu, Spark 1.4.0 Standalone

      Description

      In my spark-defaults.conf I have configured a set of libararies to be uploaded to my Spark 1.4.0 Standalone cluster. The entry appears as:

      spark.files libarary.zip,file1.py,file2.py

      When I execute spark-submit -v test.py

      I see that spark-submit reads the defaults correctly, but that it overwrites the "spark.files" default entry and replaces it with the name if the job script, i.e. "test.py".

      This behavior doesn't seem intuitive. test.py, should be added to the spark working folder, but it should not overwrite the "spark.files" defaults.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                whisperstream axel dahl
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: