[MAPREDUCE-5792] When mapreduce.jobhistory.intermediate-done-dir isn't writable, application fails with generic error

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.3.0
• Component/s: mr-am, mrv2
• Assignee: Mohammad Islam
• Reporter: Travis Thompson
• Created: 11/Mar/14 01:31
• Updated: 17/Mar/14 19:05
• Resolved: 17/Mar/14 19:05

Description

When trying to run an application and the permissions are wrong on mapreduce.jobhistory.intermediate-done-dir, the MapReduce AM fails with a non-descriptive error message:

Application application_1394227890066_0004 failed 2 times due to AM Container for appattempt_1394227890066_0004_000002 exited with exitCode: 1 due to: Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
at org.apache.hadoop.util.Shell.run(Shell.java:418)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:279)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
main : command provided 1
main : user is tthompso
main : requested yarn user is tthompso
Container exited with a non-zero exit code 1
.Failing this attempt.. Failing the application. 

When permissions are corrected on this dir, applications are able to run. There should probably be some sort of check on this dir before launching the AM so a more meaningful error message can be thrown.

Issue Links

• duplicates: YARN-675 In YarnClient, pull AM logs on AM container failure