[YARN-11039] LogAggregationFileControllerFactory::getFileControllerForRead can leak threads

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.3.4
• Fix Version/s: 3.4.0, 3.3.5
• Component/s: log-aggregation
• Labels: performance stability
• Target Version/s: 3.3.5
• Assignee: Steve Loughran
• Reporter: Rajesh Balamohan
• Created: 08/Dec/21 09:13
• Updated: 16/Dec/22 12:13
• Resolved: 18/Oct/22 15:17

Description

getFileControllerForRead::getFileControllerForRead internally opens up a new FS object everytime and is not closed.

When cloud connectors (e.g s3a) is used along with Knox, it ends up leaking KnoxTokenMonitor for every unclosed FS object causing thread leaks in NM.

Lines of interest:

https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/LogAggregationFileControllerFactory.java#L167

       try {
          Path remoteAppLogDir = fileController.getOlderRemoteAppLogDir(appId,
              appOwner);
          if (LogAggregationUtils.getNodeFiles(conf, remoteAppLogDir, appId,
              appOwner).hasNext()) {
            return fileController;
          }
        } catch (Exception ex) {
          diagnosticsMsg.append(ex.getMessage() + "\n");
          continue;
        }

https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogAggregationUtils.java#L252

	
  public static RemoteIterator<FileStatus> getNodeFiles(Configuration conf,
      Path remoteAppLogDir, ApplicationId appId, String appOwner)
      throws IOException {
    Path qualifiedLogDir =
        FileContext.getFileContext(conf).makeQualified(remoteAppLogDir);
    return FileContext.getFileContext(
        qualifiedLogDir.toUri(), conf).listStatus(remoteAppLogDir);
  }

Issue Links

• is fixed by: HADOOP-18476 Abfs and S3A FileContext bindings to close wrapped filesystems in finalizer
• is related to: YARN-11397 Memory leak when reading aggregated logs from s3 (LogAggregationTFileController::readAggregatedLogs)