[YARN-11397] Memory leak when reading aggregated logs from s3 (LogAggregationTFileController::readAggregatedLogs)

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 3.2.2
• Component/s: log-aggregation

Environment

Remote logs dir on s3.

• Assignee: Unassigned
• Reporter: Maciej Smolenski
• Created: 16/Dec/22 12:11
• Updated: 18/Dec/22 11:53

Description

Reproduction code in the attachment.

When collecting aggregated logs from s3 in a loop (see reproduction code) we can easily see that the number of 'S3AInstrumentation' is increasing although the number of 'S3AFileSystem' is not increasing. It means that 'S3AInstrumentation' is not released together with 'S3AFileSystem' as it should be. The root cause of this seems to be the missing close on S3AFileSystem.

The issue seems similar to https://issues.apache.org/jira/browse/YARN-11039 but the issue is a 'memory leak' (not a 'thread leak') and affected version is earlier here (3.2.2).

Issue Links

• depends upon: HADOOP-18476 Abfs and S3A FileContext bindings to close wrapped filesystems in finalizer
• depends upon: HADOOP-18526 Leak of S3AInstrumentation instances via hadoop Metrics references
• relates to: YARN-11039 LogAggregationFileControllerFactory::getFileControllerForRead can leak threads