Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-2764

Action failures related to HADOOP_CLASSPATH when using Hadoop 2.7.3

    Details

    • Type: Bug
    • Status: Open
    • Priority: Blocker
    • Resolution: Unresolved
    • Affects Version/s: 4.3.0
    • Fix Version/s: None
    • Component/s: action, core
    • Labels:
      None
    • Environment:

      Hadoop 2.7.3, Oozie 4.3.0, Hive 2.1.0, Pig 0.16.0, Spark 1.6.1

      Description

      Every action tested (MapReduce, Java, Spark, Hive, Pig) are failing with a "java.lang.NoSuchFieldError: HADOOP_CLASSPATH".

      Oozie has been compiled with the command:

      $ bin/mkdistro.sh -DskipTests -Phadoop-2 -Dhadoop.version=2.7.3 -Dpig.version=0.16.0 -Dspark.version=1.6.1
      

      The error stacktrace is below:

      2016-12-26T14:11:16+01:00 oozie01.bigdata.fr oozie INFO - org.apache.oozie.service.CoordMaterializeTriggerService$CoordMaterializeTriggerRunnableUSER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] CoordMaterializeTriggerService - Curr Date= 2016-12-26T13:16Z, Num jobs to materialize = 0
      2016-12-26T14:11:16+01:00 oozie01.bigdata.fr oozie INFO - org.apache.oozie.service.CoordMaterializeTriggerService$CoordMaterializeTriggerRunnableUSER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Released lock for [org.apache.oozie.service.CoordMaterializeTriggerService]
      2016-12-26T14:11:16+01:00 oozie01.bigdata.fr oozie INFO - org.apache.oozie.service.StatusTransitService$StatusTransitRunnableReleased lock for [org.apache.oozie.service.StatusTransitService]
      2016-12-26T14:11:16+01:00 oozie01.bigdata.fr oozie INFO - org.apache.oozie.command.PurgeXCommandUSER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] STARTED Purge to purge Workflow Jobs older than [30] days, Coordinator Jobs older than [7] days, and Bundlejobs older than [7] days.
      2016-12-26T14:11:16+01:00 oozie01.bigdata.fr oozie INFO - org.apache.oozie.command.PurgeXCommandUSER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] ENDED Purge deleted [0] workflows, [0] coordinatorActions, [0] coordinators, [0] bundles
      2016-12-26T14:11:16+01:00 oozie01.bigdata.fr oozie INFO - org.apache.oozie.service.PauseTransitServiceReleased lock for [org.apache.oozie.service.PauseTransitService]
      2016-12-26T14:11:17+01:00 oozie01.bigdata.fr oozie INFO - org.apache.oozie.command.wf.ActionStartXCommandUSER[shfs3453] GROUP[-] TOKEN[] APP[SparkPi-test] JOB[0000000-161226114905740-oozie-W] ACTION[0000000-161226114905740-oozie-W@spark-node] Start action [0000000-161226114905740-oozie-W@spark-node] with user-retry state : userRetryCount [0], userRetryMax [3], userRetryInterval [1]
      2016-12-26T14:11:17+01:00 oozie01.bigdata.fr oozie INFO - org.apache.oozie.service.HadoopAccessorServiceUSER[shfs3453] GROUP[-] TOKEN[] APP[SparkPi-test] JOB[0000000-161226114905740-oozie-W] ACTION[0000000-161226114905740-oozie-W@spark-node] Processing configuration file [/opt/application/Oozie/current/conf/action-conf/default.xml] for action [default] and hostPort [*]
      2016-12-26T14:11:17+01:00 oozie01.bigdata.fr oozie INFO - org.apache.oozie.service.HadoopAccessorServiceUSER[shfs3453] GROUP[-] TOKEN[] APP[SparkPi-test] JOB[0000000-161226114905740-oozie-W] ACTION[0000000-161226114905740-oozie-W@spark-node] Processing configuration file [/opt/application/Oozie/current/conf/action-conf/spark.xml] for action [spark] and hostPort [*]
      2016-12-26T14:11:22+01:00 oozie01.bigdata.fr oozie INFO - org.apache.oozie.service.HadoopAccessorServiceUSER[shfs3453] GROUP[-] TOKEN[] APP[SparkPi-test] JOB[0000000-161226114905740-oozie-W] ACTION[0000000-161226114905740-oozie-W@spark-node] Delegation Token Renewer details: Principal=rm/_HOST@SANDBOX.HADOOP,Target=sandbox-RMS:8032,Renewer=rm/sandbox-rms@SANDBOX.HADOOP
      2016-12-26T14:11:22+01:00 oozie01.bigdata.fr oozie WARN - org.apache.hadoop.mapreduce.JobResourceUploaderHadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
      2016-12-26T14:11:22+01:00 oozie01.bigdata.fr oozie WARN - org.apache.hadoop.mapreduce.JobResourceUploaderNo job jar file set.  User classes may not be found. See Job or Job#setJar(String).
      2016-12-26T14:11:23+01:00 oozie01.bigdata.fr oozie ERROR - org.apache.oozie.command.wf.ActionStartXCommandUSER[shfs3453] GROUP[-] TOKEN[] APP[SparkPi-test] JOB[0000000-161226114905740-oozie-W] ACTION[0000000-161226114905740-oozie-W@spark-node] Error,
      2016-12-26T14:11:23.573405+01:00 localhost java.lang.NoSuchFieldError: HADOOP_CLASSPATH
      2016-12-26T14:11:23.573534+01:00 localhost     at org.apache.hadoop.mapreduce.v2.util.MRApps.setClasspath(MRApps.java:248)
      2016-12-26T14:11:23.573571+01:00 localhost     at org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:458)
      2016-12-26T14:11:23.573602+01:00 localhost     at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:285)
      2016-12-26T14:11:23.573633+01:00 localhost     at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:240)
      2016-12-26T14:11:23.573662+01:00 localhost     at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
      2016-12-26T14:11:23.573691+01:00 localhost     at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
      2016-12-26T14:11:23.573720+01:00 localhost     at java.security.AccessController.doPrivileged(Native Method)
      2016-12-26T14:11:23.573754+01:00 localhost     at javax.security.auth.Subject.doAs(Subject.java:415)
      2016-12-26T14:11:23.573783+01:00 localhost     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
      2016-12-26T14:11:23.573813+01:00 localhost     at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
      2016-12-26T14:11:23.573842+01:00 localhost     at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:575)
      2016-12-26T14:11:23.573870+01:00 localhost     at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:570)
      2016-12-26T14:11:23.573899+01:00 localhost     at java.security.AccessController.doPrivileged(Native Method)
      2016-12-26T14:11:23.573928+01:00 localhost     at javax.security.auth.Subject.doAs(Subject.java:415)
      2016-12-26T14:11:23.573957+01:00 localhost     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
      2016-12-26T14:11:23.573985+01:00 localhost     at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:570)
      2016-12-26T14:11:23.574014+01:00 localhost     at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:561)
      2016-12-26T14:11:23.574047+01:00 localhost     at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1183)
      2016-12-26T14:11:23.574076+01:00 localhost     at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1369)
      2016-12-26T14:11:23.574106+01:00 localhost     at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
      2016-12-26T14:11:23.574162+01:00 localhost     at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:65)
      2016-12-26T14:11:23.574193+01:00 localhost     at org.apache.oozie.command.XCommand.call(XCommand.java:287)
      2016-12-26T14:11:23.574222+01:00 localhost     at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      2016-12-26T14:11:23.574252+01:00 localhost     at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
      2016-12-26T14:11:23.574286+01:00 localhost     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      2016-12-26T14:11:23.574341+01:00 localhost     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      2016-12-26T14:11:23.574372+01:00 localhost     at java.lang.Thread.run(Thread.java:745)
      

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              BigDataOrange Alexandre Linte
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: