Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6941

The default setting doesn't work for MapReduce job



    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Not A Problem
    • 3.0.0-beta1
    • None
    • None
    • None


      On the deployment of hadoop 3 cluster (based on current trunk branch) with default settings, the MR job will get failed as following exceptions:

      2017-08-16 13:00:03,846 INFO mapreduce.Job: Job job_1502913552390_0001 running in uber mode : false
      2017-08-16 13:00:03,847 INFO mapreduce.Job:  map 0% reduce 0%
      2017-08-16 13:00:03,864 INFO mapreduce.Job: Job job_1502913552390_0001 failed with state FAILED due to: Application application_1502913552390_0001 failed 2 times due to AM Container for appattempt_1502913552390_0001_000002 exited with  exitCode: 1
      Failing this attempt.Diagnostics: [2017-08-16 13:00:02.963]Exception from container-launch.
      Container id: container_1502913552390_0001_02_000001
      Exit code: 1
      Stack trace: ExitCodeException exitCode=1:
      	at org.apache.hadoop.util.Shell.runCommand(Shell.java:994)
      	at org.apache.hadoop.util.Shell.run(Shell.java:887)
      	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1212)
      	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:295)
      	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:455)
      	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:275)
      	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:90)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)

      This is because mapreduce related jar are not added into yarn setup by default. To make MR job run successful, we need to add following configurations to yarn-site.xml now:


      But this config is not necessary for previous version of Hadoop. We should fix this issue before beta release otherwise it will be a regression for configuration changes.

      This could be more like a YARN issue (if so, we should move), depends on how we fix it finally.


        Issue Links



              Unassigned Unassigned
              junping_du Junping Du
              0 Vote for this issue
              7 Start watching this issue