[HADOOP-18091] S3A auditing leaks memory through ThreadLocal references

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.2
• Fix Version/s: 3.3.5
• Component/s: fs/s3
• Labels: pull-request-available
• Target Version/s: 3.3.5
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 22/Jan/22 15:34
• Updated: 23/Sep/22 09:12
• Resolved: 10/Feb/22 12:35

Description

ActiveAuditManagerS3A uses thread locals to map to active audit spans, which (because they are wrapped) include back reference to the audit manager instance and the config it was created with.

these do not get cleaned up when the FS instance is closed.

if you have a long lived process creating and destroying many FS instances, then memory gets used up.

This fix moves off threadlocal into a map of weak references. while a strong reference is held` (for example in the s3a entry point method) then the references will always resolve. but if those are released then when a GC is triggered these weak references will not be retained, so not use up memory other than entries in the the ha!sh map. the map is held by the s3a auditing integration, so when the fs is closed, everything is freed up.

backport/cherrypicking

if this is backported, followup with HADOOP-18456

Issue Links

• causes: HADOOP-18456 NullPointerException in ObjectListingIterator's constructor
• depends upon: HADOOP-18094 Disable S3A auditing by default.
• is caused by: HADOOP-17511 Add an Audit plugin point for S3A auditing/context
• is related to: HADOOP-18150 Fix ITestAuditManagerDisabled after S3A audit logging was enabled in HADOOP-18091
• links to: GitHub Pull Request #3916
• links to: GitHub Pull Request #3930