[SPARK-6890] Local cluster mode is broken with SPARK_PREPEND_CLASSES

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.4.0
• Fix Version/s: 1.4.0
• Component/s: Spark Core
• Target Version/s: 1.4.0
• Assignee: Marcelo Masiero Vanzin
• Reporter: Davies Liu
• Created: 13/Apr/15 23:56
• Updated: 16/Apr/15 23:42
• Resolved: 15/Apr/15 01:52

Description

In master, local cluster mode is broken. If I run `bin/spark-submit --master local-cluster[2,1,512]`, my executors keep failing with class not found exception. It appears that the assembly jar is not added to the executors' class paths. I suspect that this is caused by https://github.com/apache/spark/pull/5085.

Exception in thread "main" java.lang.NoClassDefFoundError: scala/Option
	at java.lang.Class.getDeclaredMethods0(Native Method)
	at java.lang.Class.privateGetDeclaredMethods(Class.java:2531)
	at java.lang.Class.getMethod0(Class.java:2774)
	at java.lang.Class.getMethod(Class.java:1663)
	at sun.launcher.LauncherHelper.getMainMethod(LauncherHelper.java:494)
	at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:486)
Caused by: java.lang.ClassNotFoundException: scala.Option
	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

Issue Links

• duplicates: SPARK-6668 repeated asking to remove non-existent executor
• links to: [Github] Pull Request #5504 (vanzin)