[HIVE-13836] DbNotifications giving an error = Invalid state. Transaction has already started

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 2.1.1, 2.2.0
• Labels: patch-available
• Target Version/s: 2.1.1
• Assignee: Nachiket Vaidya
• Reporter: Nachiket Vaidya
• Created: 24/May/16 22:37
• Updated: 01/Oct/19 22:07
• Resolved: 06/Jun/16 18:03

Description

I used pyhs2 python client to create tables/partitions in hive. I was working fine until I moved to multithreaded scripts which created 8 connections and ran DDL queries concurrently.
I got the error as

2016-05-04 17:49:26,226 ERROR org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-4-thread-194]: HMSHandler Fatal error: Invalid state. Transaction has already started
org.datanucleus.transaction.NucleusTransactionException: Invalid state. Transaction has already started
        at org.datanucleus.transaction.TransactionManager.begin(TransactionManager.java:47)
        at org.datanucleus.TransactionImpl.begin(TransactionImpl.java:131)
        at org.datanucleus.api.jdo.JDOTransaction.internalBegin(JDOTransaction.java:88)
        at org.datanucleus.api.jdo.JDOTransaction.begin(JDOTransaction.java:80)
        at org.apache.hadoop.hive.metastore.ObjectStore.openTransaction(ObjectStore.java:463)
        at org.apache.hadoop.hive.metastore.ObjectStore.addNotificationEvent(ObjectStore.java:7522)
        at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
        at com.sun.proxy.$Proxy10.addNotificationEvent(Unknown Source)
        at org.apache.hive.hcatalog.listener.DbNotificationListener.enqueue(DbNotificationListener.java:261)
        at org.apache.hive.hcatalog.listener.DbNotificationListener.onCreateTable(DbNotificationListener.java:123)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1483)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1502)
        at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:138)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
        at com.sun.proxy.$Proxy14.create_table_with_environment_context(Unknown Source)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:9267)

Issue Links

• is related to: HIVE-15373 DbNotificationListener should use thread-local RawStore
• is related to: HIVE-18885 DbNotificationListener has a deadlock between Java and DB locks (2.x line)
• is related to: HIVE-14841 Replication - Phase 2
• relates to: HIVE-13869 ObjectStore synchronization issue manifested in db notification listener