[HIVE-16321] Possible deadlock in metastore with Acid enabled

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.3.0
• Fix Version/s: 2.3.0, 2.4.0, 3.0.0
• Component/s: Transactions
• Labels: TODOC2.3
• Target Version/s: 2.3.0, 2.4.0, 3.0.0
• Assignee: Eugene Koifman
• Reporter: Eugene Koifman
• Created: 28/Mar/17 21:57
• Updated: 16/Sep/21 18:49
• Resolved: 22/Apr/17 22:35

Description

TxnStore.MutexAPI is a mechanism how different Metastore instances can coordinate their operations. It uses a JDBCConnection to achieve it.

In some cases this may lead to deadlock. TxnHandler uses a connection pool of fixed size. Suppose you have X simultaneous calls to TxnHandler.lock(), where X is >= size of the pool. This take all connections form the pool, so when

handle = getMutexAPI().acquireLock(MUTEX_KEY.CheckLock.name());

is executed in TxnHandler.checkLock(Connection dbConn, long extLockId) the pool is empty and the system is deadlocked.

MutexAPI can't use the same connection as the operation it's protecting. (TxnHandler.checkLock(Connection dbConn, long extLockId) is an example).

We could make MutexAPI use a separate connection pool (size > 'primary' conn pool).

Or we could make TxnHandler.lock(LockRequest rqst) return immediately after enqueueing the lock with the expectation that the caller will always follow up with a call to checkLock(CheckLockRequest rqst).

cc f1sherox

Issue Links

• relates to: HIVE-13842 Expose ability to set number of connections in the pool in TxnHandler
• relates to: HIVE-16383 Switch to HikariCP as default connection pooling
• relates to: HIVE-25522 NullPointerException in TxnHandler