Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27289

spark-submit explicit configuration does not take effect but Spark UI shows it's effective

    Details

      Description

      The doc says that  "In general, configuration values explicitly set on a SparkConf take the highest precedence, then flags passed to spark-submit, then values in the defaults file", but when setting spark.local.dir through --conf with spark-submit, it still uses the values from ${SPARK_HOME}/conf/spark-defaults.conf, what's more, the Spark runtime UI environment variables shows the value from --conf, which is really misleading.

      e.g.

      I set submit my application through the command:

      /opt/spark233/bin/spark-submit --properties-file /opt/spark.conf --conf spark.local.dir=/tmp/spark_local -v --class org.apache.spark.examples.mllib.SparseNaiveBayes --master spark://bdw-slave20:7077 /opt/sparkbench/assembly/target/sparkbench-assembly-7.1-SNAPSHOT-dist.jar hdfs://bdw-slave20:8020/Bayes/Input

       

      the spark.local.dir in ${SPARK_HOME}/conf/spark-defaults.conf is:

      spark.local.dir=/mnt/nvme1/spark_local

      when the application is running, I found the intermediate shuffle data was wrote to /mnt/nvme1/spark_local, which is set through ${SPARK_HOME}/conf/spark-defaults.conf, but the Web UI shows that the environment value spark.local.dir=/tmp/spark_local.

      The spark-submit verbose also shows spark.local.dir=/tmp/spark_local, it's misleading. 

       

      Unable to render embedded object: File (image-2019-03-27-10-59-38-377.png) not found.

      spark-submit verbose:

      XXXX

      Spark properties used, including those specified through
      --conf and those from the properties file /opt/spark.conf:
      (spark.local.dir,/tmp/spark_local)
      (spark.default.parallelism,132)
      (spark.driver.memory,10g)
      (spark.executor.memory,352g)

      XXXXX

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              KaiXu KaiXu
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: