Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-23635

Spark executor env variable is overwritten by same name AM env variable

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.4.0
    • Component/s: YARN
    • Labels:
      None

      Description

      In the current Spark on YARN code, AM always will copy and overwrite its env variables to executors, so we cannot set different values for executors.

      To reproduce issue, user could start spark-shell like:

      ./bin/spark-shell --master yarn-client --conf spark.executorEnv.SPARK_ABC=executor_val --conf  spark.yarn.appMasterEnv.SPARK_ABC=am_val
      
      

      Then check executor env variables by

      sc.parallelize(1 to 1).flatMap \{ i => sys.env.toSeq }.collect.foreach(println)
      
      

      You will always get am_val instead of executor_val. So we should not let AM to overwrite specifically set executor env variables.

        Attachments

          Activity

            People

            • Assignee:
              jerryshao Saisai Shao
              Reporter:
              jerryshao Saisai Shao
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: