Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
4.2.0
-
None
-
None
-
None
Description
I used spark-examples-1.5.1-hadoop2.6.0.jar to generate a test spark action in Oozie, it succeeded sometimes, but in most of the times, it failed.
I checked the Hadoop job history, and it said
ERROR [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
java.lang.NoSuchMethodError: org.apache.hadoop.mapred.TaskLog.createLogSyncer()Ljava/util/concurrent/ScheduledExecutorService;
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.<init>(MRAppMaster.java:244)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.<init>(MRAppMaster.java:227)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1412)
2015-10-22 17:01:56,203 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting with status 1
I'm using Hadoop 2.6.0, Spark 1.5.1, Oozie 4.2.0, and I put spark-assembly-1.5.1-hadoop2.6.0.jar together with the example jar in the lib, and my workflow.xml is as followings.
<action name="sparkTest"> <spark xmlns="uri:oozie:spark-action:0.1"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <master>yarn-cluster</master> <name>SparkPi</name> <class>org.apache.spark.examples.JavaSparkPi</class> <jar>${nameNode}/sparkOozieTest/workflow/lib/spark-examples-1.5.1-hadoop2.6.0.jar</jar> <arg>10</arg> </spark> <ok to="end"/> <error to="fail"/> </action>
As it indeed succeeded sometimes, I don't think there is anything wrong with my configs. So is it an Oozie bug or is there anything I missed?