[SPARK-12345] Mesos cluster mode is broken

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.6.0
• Fix Version/s: 1.6.0
• Component/s: Mesos
• Target Version/s: 1.6.0
• Assignee: Timothy Chen
• Reporter: Andrew Or
• Created: 15/Dec/15 19:10
• Updated: 03/Nov/18 19:21
• Resolved: 16/Dec/15 19:01

Description

The same setup worked in 1.5.2 but is now failing for 1.6.0-RC2.

The driver is confused about where SPARK_HOME is. It resolves `mesos.executor.uri` or `spark.mesos.executor.home` relative to the filesystem where the driver runs, which is wrong.

I1215 15:00:39.411212 28032 exec.cpp:134] Version: 0.25.0
I1215 15:00:39.413512 28037 exec.cpp:208] Executor registered on slave 130bdc39-44e7-4256-8c22-602040d337f1-S1
bin/spark-submit: line 27: /Users/dragos/workspace/Spark/dev/rc-tests/spark-1.6.0-bin-hadoop2.6/bin/spark-class: No such file or directory

Issue Links

• breaks: SPARK-12413 Mesos ZK persistence throws a NotSerializableException
• is related to: SPARK-25934 Mesos: SPARK_CONF_DIR should not be propogated by spark submit
• is related to: SPARK-11759 Spark task on mesos with docker fails with sh: 1: /opt/spark/bin/spark-class: not found
• links to: [Github] Pull Request #10315 (dragos)
• links to: [Github] Pull Request #10329 (skyluc)
• links to: [Github] Pull Request #10332 (tnachen)
• links to: [Github] Pull Request #10359 (dragos)