[YARN-11039] LogAggregationFileControllerFactory::getFileControllerForRead can leak threads - ASF JIRA

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Blocker
Resolution: Fixed
Affects Version/s: 3.3.4
Fix Version/s: 3.4.0, 3.3.5
Component/s: log-aggregation
Labels:
- performance
- stability

Target Version/s:

3.3.5

Description

getFileControllerForRead::getFileControllerForRead internally opens up a new FS object everytime and is not closed.

When cloud connectors (e.g s3a) is used along with Knox, it ends up leaking KnoxTokenMonitor for every unclosed FS object causing thread leaks in NM.

Lines of interest:

https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/LogAggregationFileControllerFactory.java#L167

       try {
          Path remoteAppLogDir = fileController.getOlderRemoteAppLogDir(appId,
              appOwner);
          if (LogAggregationUtils.getNodeFiles(conf, remoteAppLogDir, appId,
              appOwner).hasNext()) {
            return fileController;
          }
        } catch (Exception ex) {
          diagnosticsMsg.append(ex.getMessage() + "\n");
          continue;
        }

https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogAggregationUtils.java#L252

	
  public static RemoteIterator<FileStatus> getNodeFiles(Configuration conf,
      Path remoteAppLogDir, ApplicationId appId, String appOwner)
      throws IOException {
    Path qualifiedLogDir =
        FileContext.getFileContext(conf).makeQualified(remoteAppLogDir);
    return FileContext.getFileContext(
        qualifiedLogDir.toUri(), conf).listStatus(remoteAppLogDir);
  }