Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-18067 Über-jira: S3A Hadoop 3.3.4 features
  3. HADOOP-18091

S3A auditing leaks memory through ThreadLocal references

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.2
    • 3.3.9
    • fs/s3

    Description

      ActiveAuditManagerS3A uses thread locals to map to active audit spans, which (because they are wrapped) include back reference to the audit manager instance and the config it was created with.

      these do not get cleaned up when the FS instance is closed.

      if you have a long lived process creating and destroying many FS instances, then memory gets used up.

      This fix moves off threadlocal into a map of weak references. while a strong reference is held` (for example in the s3a entry point method) then the references will always resolve. but if those are released then when a GC is triggered these weak references will not be retained, so not use up memory other than entries in the the ha!sh map. the map is held by the s3a auditing integration, so when the fs is closed, everything is freed up.

      Attachments

        Issue Links

          Activity

            People

              stevel@apache.org Steve Loughran
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 7h 40m
                  7h 40m