Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-11039

LogAggregationFileControllerFactory::getFileControllerForRead should close FS

    XMLWordPrintableJSON

Details

    Description

      getFileControllerForRead::getFileControllerForRead internally opens up a new FS object everytime and is not closed.

      When cloud connectors (e.g s3a) is used along with Knox, it ends up leaking KnoxTokenMonitor for every unclosed FS object causing thread leaks in NM.

      Lines of interest:

      https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/LogAggregationFileControllerFactory.java#L167

             try {
                Path remoteAppLogDir = fileController.getOlderRemoteAppLogDir(appId,
                    appOwner);
                if (LogAggregationUtils.getNodeFiles(conf, remoteAppLogDir, appId,
                    appOwner).hasNext()) {
                  return fileController;
                }
              } catch (Exception ex) {
                diagnosticsMsg.append(ex.getMessage() + "\n");
                continue;
              }
      

      https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogAggregationUtils.java#L252

      	
        public static RemoteIterator<FileStatus> getNodeFiles(Configuration conf,
            Path remoteAppLogDir, ApplicationId appId, String appOwner)
            throws IOException {
          Path qualifiedLogDir =
              FileContext.getFileContext(conf).makeQualified(remoteAppLogDir);
          return FileContext.getFileContext(
              qualifiedLogDir.toUri(), conf).listStatus(remoteAppLogDir);
        }
      

      Attachments

        Activity

          People

            Unassigned Unassigned
            rajesh.balamohan Rajesh Balamohan
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: