Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-2389

Spark action failed with error starting MRAppMaster

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 4.2.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      I used spark-examples-1.5.1-hadoop2.6.0.jar to generate a test spark action in Oozie, it succeeded sometimes, but in most of the times, it failed.
      I checked the Hadoop job history, and it said

      ERROR [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
      java.lang.NoSuchMethodError: org.apache.hadoop.mapred.TaskLog.createLogSyncer()Ljava/util/concurrent/ScheduledExecutorService;
      at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.<init>(MRAppMaster.java:244)
      at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.<init>(MRAppMaster.java:227)
      at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1412)
      2015-10-22 17:01:56,203 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting with status 1

      I'm using Hadoop 2.6.0, Spark 1.5.1, Oozie 4.2.0, and I put spark-assembly-1.5.1-hadoop2.6.0.jar together with the example jar in the lib, and my workflow.xml is as followings.

          <action name="sparkTest">
              <spark xmlns="uri:oozie:spark-action:0.1">
                  <job-tracker>${jobTracker}</job-tracker>
                  <name-node>${nameNode}</name-node>
                  <master>yarn-cluster</master>
                  <name>SparkPi</name>
                  <class>org.apache.spark.examples.JavaSparkPi</class>
                  <jar>${nameNode}/sparkOozieTest/workflow/lib/spark-examples-1.5.1-hadoop2.6.0.jar</jar>
                  <arg>10</arg>
              </spark>
              <ok to="end"/>
              <error to="fail"/>
          </action>
      

      As it indeed succeeded sometimes, I don't think there is anything wrong with my configs. So is it an Oozie bug or is there anything I missed?

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              hunttang Hunt Tang
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated: