[HIVE-20600] Metastore connection leak

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.3.3
• Component/s: HiveServer2
• Assignee: Unassigned
• Reporter: Damon Cortesi
• Created: 19/Sep/18 17:22
• Updated: 25/Aug/20 16:29

Description

Within the execute method of HiveServer2, there appears to be a connection leak. With fairly straightforward series of INSERT statements, the connection count in the logs continues to increase over time. Under certain loads, this can also consume all underlying threads of the Hive metastore and result in HS2 becoming unresponsive to new connections.

The log below is the result of some python code executing a single insert statement, and then looping through a series of 10 more insert statements. We can see there's one dangling connection left open after each execution leaving us with 12 open connections (11 from the execute statements + 1 from HS2 startup).

2018-09-19T17:14:32,108 INFO [main([])]: hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to metastore, current connections: 1
 2018-09-19T17:14:48,175 INFO [29049f74-73c4-4f48-9cf7-b4bfe524a85b HiveServer2-Handler-Pool: Thread-31([])]: hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to metastore, current connections: 2
 2018-09-19T17:15:05,543 INFO [HiveServer2-Background-Pool: Thread-36([])]: hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to metastore, current connections: 1
 2018-09-19T17:15:05,548 INFO [HiveServer2-Background-Pool: Thread-36([])]: hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to metastore, current connections: 2
 2018-09-19T17:15:05,932 INFO [HiveServer2-Background-Pool: Thread-36([])]: hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to metastore, current connections: 1
 2018-09-19T17:15:05,935 INFO [HiveServer2-Background-Pool: Thread-36([])]: hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to metastore, current connections: 2
 2018-09-19T17:15:06,123 INFO [HiveServer2-Background-Pool: Thread-36([])]: hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to metastore, current connections: 1
 2018-09-19T17:15:06,126 INFO [HiveServer2-Background-Pool: Thread-36([])]: hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to metastore, current connections: 2
...
 2018-09-19T17:15:20,626 INFO [29049f74-73c4-4f48-9cf7-b4bfe524a85b HiveServer2-Handler-Pool: Thread-31([])]: hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to metastore, current connections: 12
 2018-09-19T17:15:21,153 INFO [HiveServer2-Background-Pool: Thread-162([])]: hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to metastore, current connections: 11
 2018-09-19T17:15:21,155 INFO [HiveServer2-Background-Pool: Thread-162([])]: hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to metastore, current connections: 12
 2018-09-19T17:15:21,306 INFO [HiveServer2-Background-Pool: Thread-162([])]: hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to metastore, current connections: 11
 2018-09-19T17:15:21,308 INFO [HiveServer2-Background-Pool: Thread-162([])]: hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to metastore, current connections: 12
 2018-09-19T17:15:21,385 INFO [HiveServer2-Background-Pool: Thread-162([])]: hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to metastore, current connections: 11
 2018-09-19T17:15:21,387 INFO [HiveServer2-Background-Pool: Thread-162([])]: hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to metastore, current connections: 12
 2018-09-19T17:15:21,541 INFO [HiveServer2-Handler-Pool: Thread-31([])]: hive.metastore (HiveMetaStoreClient.java:open(481)) - Opened a connection to metastore, current connections: 13
 2018-09-19T17:15:21,542 INFO [HiveServer2-Handler-Pool: Thread-31([])]: hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to metastore, current connections: 12

Attached is a simple impyla script that triggers the condition.