[HIVE-22365] "MetaException: Couldn't acquire the DB log notification lock because we reached the maximum # of retries" during metadata scale tests

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Assignee: mahesh kumar behera
• Reporter: mahesh kumar behera
• Created: 18/Oct/19 04:47
• Updated: 18/Oct/19 04:55
• Resolved: 18/Oct/19 04:55

Description

The issue is because of the leaked open transaction in Objectstore::getPartition function. Here if jdo throws some exception during convertToPart then commit is not done.
openTransaction();
MTable table = this.getMTable(catName, dbName, tableName);
MPartition mpart = getMPartition(catName, dbName, tableName, part_vals);
Partition part = convertToPart(mpart);
commitTransaction();
 

Because of this, all subsequent transactions of this thread are not committed.

if ((openTrasactionCalls == 0) && currentTransaction.isActive()) {
  transactionStatus = TXN_STATUS.COMMITED;
  currentTransaction.commit();
} 

This is causing the select for update lock on NOTIFICATION_SEQUENCE to never be released and all other threads are failing to get this lock and timing out.

So the fix to do the operation is a try-catch block and rollback the txn in case of failure.

 

Issue Links

• is duplicated by: HIVE-16839 Unbalanced calls to openTransaction/commitTransaction when alter the same partition concurrently