[HDFS-3545] DFSClient leak due to malfunctioning of FileSystem Cache

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 2.0.0-alpha, 3.0.0-alpha1
• Component/s: hdfs-client
• Assignee: Unassigned
• Reporter: amith
• Created: 19/Jun/12 10:59
• Updated: 30/Oct/17 17:08

Description

For every FileSystem.get new FileSystem object is getting created even though the UGI object passed has same name. This is creating the lot of FileSystem objects and cached in FileSystem cache instead of using the same cached object .

This is causing the Cache to grow in size causing OOME

This behaviour can be seen in Mapred and Hive components also since they use FileSystem.get in the described fashion

Issue Links

• is related to: MAPREDUCE-4340 Node Manager leaks socket connections connected to Data Node
• is related to: HIVE-3155 Memory leak in Hive
• is related to: HIVE-4501 HS2 memory leak - FileSystem objects in FileSystem.CACHE
• is related to: SPARK-22374 STS ran into OOM in a secure cluster
• is related to: HIVE-3098 Memory leak from large number of FileSystem instances in FileSystem.CACHE