Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3925

ContainerLogsUtils#getContainerLogFile fails to read container log files from full disks.

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      ContainerLogsUtils#getContainerLogFile fails to read files from full disks.
      getContainerLogFile depends on LocalDirsHandlerService#getLogPathToRead to get the log file, but LocalDirsHandlerService#getLogPathToRead calls logDirsAllocator.getLocalPathToRead and logDirsAllocator uses configuration YarnConfiguration.NM_LOG_DIRS, which will be updated to not include full disks in LocalDirsHandlerService#checkDirs:

          Configuration conf = getConfig();
          List<String> localDirs = getLocalDirs();
          conf.setStrings(YarnConfiguration.NM_LOCAL_DIRS,
                          localDirs.toArray(new String[localDirs.size()]));
          List<String> logDirs = getLogDirs();
          conf.setStrings(YarnConfiguration.NM_LOG_DIRS,
                            logDirs.toArray(new String[logDirs.size()]));
      

      ContainerLogsUtils#getContainerLogFile is used by NMWebServices#getLogs and ContainerLogsPage.ContainersLogsBlock#render to read the log.

      Attachments

        1. YARN-3925.000.patch
          11 kB
          Zhihai Xu
        2. YARN-3925.001.patch
          7 kB
          Zhihai Xu

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            zxu Zhihai Xu
            zxu Zhihai Xu
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment