[MAPREDUCE-7094] LocalDistributedCacheManager leaves classloaders open, which leaks FDs

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.6.0
• Fix Version/s: 3.2.0
• Hadoop Flags: Reviewed
• Assignee: Ádám Szita
• Reporter: Ádám Szita
• Created: 08/May/18 15:31
• Updated: 26/Feb/20 05:29
• Resolved: 17/May/18 18:18

Description

When a user starts a local mapred task from Hive's beeline, it will leave open file descriptors on the HS2 process (which runs the mapred task).

I debugged this and saw that it is caused by LocalDistributedCacheManager class, which creates a new URLClassLoader, with a classpath for the two jars seen below. Somewhere down the line Loaders will be created in this URLClassLoader for these files effectively creating the FD's on the OS level.

This is never cleaned up after execution, although LocalDistributedCacheManager removes the files, it will not close the ClassLoader, so FDs are left open although they point to deleted files at that time:

[root@host-1 ~]# lsof -p 14439 | grep hadoop-hive
java    14439 hive  DEL       REG                8,1             3348748 /tmp/hadoop-hive/mapred/local/1525789796610/hive-exec-core.jar
java    14439 hive  DEL       REG                8,1             3348750 /tmp/hadoop-hive/mapred/local/1525789796609/hive-exec-1.1.0-cdh5.13.4-SNAPSHOT-core.jar
java    14439 hive  649r      REG                8,1   8112438   3348750 /tmp/hadoop-hive/mapred/local/1525789796609/hive-exec-1.1.0-cdh5.13.4-SNAPSHOT-core.jar (deleted)
java    14439 hive  650r      REG                8,1   8112438   3348748 /tmp/hadoop-hive/mapred/local/1525789796610/hive-exec-core.jar (deleted)