Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-7737

prelaunch.err file not found exception on container failure

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.1.0, 2.9.1, 3.0.1
    • 3.1.0, 2.10.0, 2.9.1, 3.0.1
    • None
    • None
    • Reviewed

    Description

      Hit this exception when a container failed:

      2018-01-11 19:04:08,036 ERROR org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Failed to get tail of the container's prelaunch error log file
      java.io.FileNotFoundException: File /grid/b/tmp/userlogs/application_1515190594800_1766/container_e39_1515190594800_1766_01_000002/prelaunch.err does not exist
              at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:641)
              at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:930)
              at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)
              at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.handleContainerExitWithFailure(ContainerLaunch.java:545)
              at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.handleContainerExitCode(ContainerLaunch.java:511)
              at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:319)
              at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:93)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)

      containerLogDir is picked on container launch via LocalDirAllocator#getLocalPathForWrite, which is where it looks for prelaunch.err when the container fails. But prelaunch.err (and prelaunch.out) are created in the first log dir (in ContainerLaunch#call:

              exec.writeLaunchEnv(containerScriptOutStream, environment,
                  localResources, launchContext.getCommands(),
                  new Path(containerLogDirs.get(0)), user);

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            oliverhuhuhu@gmail.com Keqiu Hu Assign to me
            jhung Jonathan Hung
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment