[YARN-7737] prelaunch.err file not found exception on container failure

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0, 2.9.1, 3.0.1
• Fix Version/s: 3.1.0, 2.10.0, 2.9.1, 3.0.1
• Hadoop Flags: Reviewed
• Assignee: Keqiu Hu
• Reporter: Jonathan Hung
• Created: 11/Jan/18 19:35
• Updated: 24/Jan/18 21:16
• Resolved: 24/Jan/18 21:16

Description

Hit this exception when a container failed:

2018-01-11 19:04:08,036 ERROR org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Failed to get tail of the container's prelaunch error log file
java.io.FileNotFoundException: File /grid/b/tmp/userlogs/application_1515190594800_1766/container_e39_1515190594800_1766_01_000002/prelaunch.err does not exist
        at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:641)
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:930)
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.handleContainerExitWithFailure(ContainerLaunch.java:545)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.handleContainerExitCode(ContainerLaunch.java:511)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:319)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:93)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

containerLogDir is picked on container launch via LocalDirAllocator#getLocalPathForWrite, which is where it looks for prelaunch.err when the container fails. But prelaunch.err (and prelaunch.out) are created in the first log dir (in ContainerLaunch#call:

        exec.writeLaunchEnv(containerScriptOutStream, environment,
            localResources, launchContext.getCommands(),
            new Path(containerLogDirs.get(0)), user);