[HIVE-4501] HS2 memory leak - FileSystem objects in FileSystem.CACHE

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 0.11.0
• Fix Version/s: 0.13.0
• Component/s: HiveServer2
• Assignee: Vaibhav Gumashta Assign to me
• Reporter: Thejas Nair
• Created: 06/May/13 04:23
• Updated: 30/Dec/14 23:11
• Resolved: 14/Mar/14 18:17

Description

org.apache.hadoop.fs.FileSystem objects are getting accumulated in FileSystem.CACHE, with HS2 in unsecure mode.

As a workaround, it is possible to set fs.hdfs.impl.disable.cache and fs.file.impl.disable.cache to true.
Users should not have to bother with this extra configuration.

As a workaround disable impersonation by setting hive.server2.enable.doAs to false.

Issue Links

• duplicates: HIVE-6312 doAs with plain sasl auth should be session aware
• is duplicated by: HIVE-8369 SimpleFetchOptimizer needs to re-enable FS caching before scanning dirs
• is related to: HIVE-5268 HiveServer2 accumulates orphaned OperationHandle objects when a client fails while executing query
• is related to: HIVE-3098 Memory leak from large number of FileSystem instances in FileSystem.CACHE
• is related to: HIVE-5296 Memory leak: OOM Error after multiple open/closed JDBC connections.
• is related to: HIVE-9234 HiveServer2 leaks FileSystem objects in FileSystem.CACHE
• relates to: HDFS-3545 DFSClient leak due to malfunctioning of FileSystem Cache
• relates to: HIVE-6484 HiveServer2 doAs should be session aware both for secured and unsecured session implementation.