[HADOOP-18476] Abfs and S3A FileContext bindings to close wrapped filesystems in finalizer

• Type: Sub-task
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.3.4
• Fix Version/s: 3.3.5, 3.3.9
• Component/s: fs/azure, fs/s3
• Labels: pull-request-available
• Target Version/s: 3.3.5
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 04/Oct/22 14:26
• Updated: 18/Dec/22 11:53
• Resolved: 18/Oct/22 15:16

Description

if you use the FileContext APIs to talk to abfs or s3a, it creates a new wrapped FileSystem implementation, and, because there is no close() call, never cleans up.

proposed: add finalizers for these two classes, which we know create helper threads, especially if plugins are added

Issue Links

• fixes: YARN-11039 LogAggregationFileControllerFactory::getFileControllerForRead can leak threads
• is depended upon by: YARN-11397 Memory leak when reading aggregated logs from s3 (LogAggregationTFileController::readAggregatedLogs)
• links to: GitHub Pull Request #4966