[SPARK-13258] --conf properties not honored in Mesos cluster mode

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 1.6.0
• Component/s: Mesos
• Assignee: Unassigned
• Reporter: Michael Gummelt
• Created: 10/Feb/16 00:54
• Updated: 31/Aug/17 13:47
• Resolved: 31/Aug/17 13:47

Description

Spark properties set on spark-submit via the deprecated SPARK_JAVA_OPTS are passed along to the driver, but those set via the preferred --conf are not.

For example, this results in the URI being fetched in the executor:

SPARK_JAVA_OPTS="-Dspark.mesos.uris=https://raw.githubusercontent.com/mesosphere/spark/master/README.md -Dspark.mesos.executor.docker.image=mesosphere/spark:1.6.0" ./bin/spark-submit --deploy-mode cluster --master mesos://10.0.78.140:7077 --class org.apache.spark.examples.SparkPi http://downloads.mesosphere.com.s3.amazonaws.com/assets/spark/spark-examples_2.10-1.5.0.jar

This does not:

SPARK_JAVA_OPTS="-Dspark.mesos.executor.docker.image=mesosphere/spark:1.6.0" ./bin/spark-submit --deploy-mode cluster --master mesos://10.0.78.140:7077 --conf spark.mesos.uris=https://raw.githubusercontent.com/mesosphere/spark/master/README.md --class org.apache.spark.examples.SparkPi http://downloads.mesosphere.com.s3.amazonaws.com/assets/spark/spark-examples_2.10-1.5.0.jar

https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L369
In the above line of code, you can see that SPARK_JAVA_OPTS is passed along to the driver, so those properties take effect.

https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L373
Whereas in this line of code, you see that --conf variables are set on SPARK_EXECUTOR_OPTS, which AFAICT has absolutely no effect because this env var is being set on the driver, not the executor.

Issue Links

• duplicates: SPARK-11327 spark-dispatcher doesn't pass along some spark properties
• relates to: SPARK-14453 Remove SPARK_JAVA_OPTS environment variable