[OOZIE-1683] UserGroupInformationService should close any filesystems opened by it

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: trunk
• Fix Version/s: 4.1.0
• Assignee: Robert Kanter
• Reporter: Robert Kanter
• Created: 29/Jan/14 18:25
• Updated: 09/Dec/14 00:45
• Resolved: 30/Jan/14 21:16

Description

When you create a FileSystem object in HDFS, it caches it.  However, when using a UGI to get the FileSystem, it caches it to that object, not its “value”.  In other words, if you create two UGIs for foo to impersonate bar, then they will each create an entry in the cache.  In Oozie, we created a UserGroupInformationService class that basically sits on top of the HDFS cache and caches the UGI objects for impersonating a user so we can reuse the same FileSystem object each time.  However, when running unit tests, we typically destroy and recreate the services for each test (sometimes even more) so Oozie’s cache was being lost and we were effectively creating a new FileSystem object all the time without closing it.

Looking at a heap dump after running many of the tests, we saw that there were 800+ FileSystem objects open for the same FileSystem.