Details
Major Description
Some discussion here: http://apache-spark-user-list.1001560.n3.nabble.com/java-lang-ClassNotFoundException-spark-on-mesos-td3510.html
Fix here (which is probably not the right fix): https://github.com/apache/spark/pull/620
This was broken in v0.9.0, was fixed in v0.9.1 and is now broken again.
Error in Mesos executor stderr:
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0502 17:31:42.672224 14688 exec.cpp:131] Version: 0.18.0
I0502 17:31:42.674959 14707 exec.cpp:205] Executor registered on slave 20140501-182306-16842879-5050-10155-0
14/05/02 17:31:42 INFO MesosExecutorBackend: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
14/05/02 17:31:42 INFO MesosExecutorBackend: Registered with Mesos as executor ID 20140501-182306-16842879-5050-10155-0
14/05/02 17:31:43 INFO SecurityManager: Changing view acls to: vagrant
14/05/02 17:31:43 INFO SecurityManager: SecurityManager, is authentication enabled: false are ui acls enabled: false users with view permissions: Set(vagrant)
14/05/02 17:31:43 INFO Slf4jLogger: Slf4jLogger started
14/05/02 17:31:43 INFO Remoting: Starting remoting
14/05/02 17:31:43 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://spark@localhost:50843]
14/05/02 17:31:43 INFO Remoting: Remoting now listens on addresses: [akka.tcp://spark@localhost:50843]
java.lang.ClassNotFoundException: org/apache/spark/serializer/JavaSerializer
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at org.apache.spark.SparkEnv$.instantiateClass$1(SparkEnv.scala:165)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:176)
at org.apache.spark.executor.Executor.<init>(Executor.scala:106)
at org.apache.spark.executor.MesosExecutorBackend.registered(MesosExecutorBackend.scala:56)
Exception in thread "Thread-0" I0502 17:31:43.710039 14707 exec.cpp:412] Deactivating the executor libprocess
The problem is that it can't find the class.