[HIVE-16455] ADD JAR command leaks JAR Files

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: HiveServer2
• Assignee: Aihua Xu
• Reporter: Aihua Xu
• Created: 14/Apr/17 20:23
• Updated: 27/Oct/18 16:44
• Resolved: 25/Sep/17 18:11

Description

HiveServer2 is leaking file handles when using ADD JAR statement and the JAR file added is not used in the query itself.

beeline> !connect jdbc:hive2://localhost:10000 admin
0: jdbc:hive2://localhost:10000> create table test_leak (a int);
0: jdbc:hive2://localhost:10000> insert into test_leak Values (1);

-- Exit beeline terminal; Find PID of HiveServer2

[root@host-10-17-80-111 ~]# lsof -p 29588 | grep "(deleted)" | wc -l
0
[root@host-10-17-80-111 ~]# beeline -u jdbc:hive2://localhost:10000/default -n admin

And run the command "ADD JAR hdfs:///tmp/hive-contrib.jar; select * from test_leak"
[root@host-10-17-80-111 ~]# lsof -p 29588 | grep "(deleted)" | wc -l
1

java    29588 hive  391u   REG              252,3    125987  2099944 /tmp/57d98f5b-1e53-44e2-876b-6b4323ac24db_resources/hive-contrib.jar (deleted)
java    29588 hive  392u   REG              252,3    125987  2099946 /tmp/eb3184ad-7f15-4a77-a10d-87717ae634d1_resources/hive-contrib.jar (deleted)
java    29588 hive  393r   REG              252,3    125987  2099825 /tmp/e29dccfc-5708-4254-addb-7a8988fc0500_resources/hive-contrib.jar (deleted)
java    29588 hive  394r   REG              252,3    125987  2099833 /tmp/5153dd4a-a606-4f53-b02c-d606e7e56985_resources/hive-contrib.jar (deleted)
java    29588 hive  395r   REG              252,3    125987  2099827 /tmp/ff3cdb05-917f-43c0-830a-b293bf397a23_resources/hive-contrib.jar (deleted)
java    29588 hive  396r   REG              252,3    125987  2099822 /tmp/60531b66-5985-421e-8eb5-eeac31fdf964_resources/hive-contrib.jar (deleted)
java    29588 hive  397r   REG              252,3    125987  2099831 /tmp/78878921-455c-438c-9735-447566ed8381_resources/hive-contrib.jar (deleted)
java    29588 hive  399r   REG              252,3    125987  2099835 /tmp/0e5d7990-30cc-4248-9058-587f7f1ff211_resources/hive-contrib.jar (deleted)

You can see the the session directory (and therefore anything in it) is set to delete only on exit.

Issue Links

• Blocked: HIVE-20824 HiveServer2 is still leaking file handles when using ADD JAR
• duplicates: HIVE-11878 ClassNotFoundException can possibly occur if multiple jars are registered one at a time in Hive