[SPARK-23635] Spark executor env variable is overwritten by same name AM env variable

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.4.0
• Component/s: Spark Core, YARN
• Assignee: Saisai Shao
• Reporter: Saisai Shao
• Created: 09/Mar/18 02:21
• Updated: 17/May/20 18:14
• Resolved: 16/Mar/18 08:23

Description

In the current Spark on YARN code, AM always will copy and overwrite its env variables to executors, so we cannot set different values for executors.

To reproduce issue, user could start spark-shell like:

./bin/spark-shell --master yarn-client --conf spark.executorEnv.SPARK_ABC=executor_val --conf  spark.yarn.appMasterEnv.SPARK_ABC=am_val

Then check executor env variables by

sc.parallelize(1 to 1).flatMap \{ i => sys.env.toSeq }.collect.foreach(println)

You will always get am_val instead of executor_val. So we should not let AM to overwrite specifically set executor env variables.

Issue Links

• links to: [Github] Pull Request #20799 (jerryshao)