Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-7737

prelaunch.err file not found exception on container failure

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.1.0, 2.9.1, 3.0.1
    • Fix Version/s: 3.1.0, 2.10.0, 2.9.1, 3.0.1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Hit this exception when a container failed:

      2018-01-11 19:04:08,036 ERROR org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Failed to get tail of the container's prelaunch error log file
      java.io.FileNotFoundException: File /grid/b/tmp/userlogs/application_1515190594800_1766/container_e39_1515190594800_1766_01_000002/prelaunch.err does not exist
              at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:641)
              at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:930)
              at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)
              at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.handleContainerExitWithFailure(ContainerLaunch.java:545)
              at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.handleContainerExitCode(ContainerLaunch.java:511)
              at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:319)
              at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:93)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

      containerLogDir is picked on container launch via LocalDirAllocator#getLocalPathForWrite, which is where it looks for prelaunch.err when the container fails. But prelaunch.err (and prelaunch.out) are created in the first log dir (in ContainerLaunch#call:

              exec.writeLaunchEnv(containerScriptOutStream, environment,
                  localResources, launchContext.getCommands(),
                  new Path(containerLogDirs.get(0)), user);

        Attachments

          Activity

            People

            • Assignee:
              oliverhuhuhu@gmail.com Keqiu Hu
              Reporter:
              jhung Jonathan Hung
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: