[HIVE-25113] Connection starvation in TxnHandler.getValidWriteIds

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: Transactions
• Labels: pull-request-available
• Target Version/s: 4.0.0
• Assignee: Yu-Wen Lai
• Reporter: Yu-Wen Lai
• Created: 13/May/21 18:33
• Updated: 25/May/21 16:33
• Resolved: 25/May/21 16:33

Description

 

The current code looks like below.

dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
validTxnList = TxnUtils.createValidReadTxnList(getOpenTxns(), 0);

In the function getOpenTxns, it will request another connection from pool. That is, this thread already held a connection, however, it would request for another connection. When there are more than 10 (default connection pool size) simultaneous getValidWriteIds requests, it can cause a starvation problem. In that situation, each thread holds a connection and waits for another connection. Then, we will see the following exception after timeout.

metastore.RetryingHMSHandler: MetaException(message:Unable to select from transaction database, java.sql.SQLTransientConnectionException: HikariPool-3 - Connection is not available, request timed out after 30000ms.

 

Issue Links

• links to: GitHub Pull Request #2272