Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-1683

UserGroupInformationService should close any filesystems opened by it

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: trunk
    • Fix Version/s: 4.1.0
    • Component/s: None
    • Labels:
      None

      Description

      When you create a FileSystem object in HDFS, it caches it.  However, when using a UGI to get the FileSystem, it caches it to that object, not its “value”.  In other words, if you create two UGIs for foo to impersonate bar, then they will each create an entry in the cache.  In Oozie, we created a UserGroupInformationService class that basically sits on top of the HDFS cache and caches the UGI objects for impersonating a user so we can reuse the same FileSystem object each time.  However, when running unit tests, we typically destroy and recreate the services for each test (sometimes even more) so Oozie’s cache was being lost and we were effectively creating a new FileSystem object all the time without closing it.

      Looking at a heap dump after running many of the tests, we saw that there were 800+ FileSystem objects open for the same FileSystem.

        Attachments

        1. OOZIE-1683.patch
          2 kB
          Robert Kanter

          Activity

            People

            • Assignee:
              rkanter Robert Kanter
              Reporter:
              rkanter Robert Kanter
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: