[HIVE-23707] Unable to create materialized views with transactions enabled with MySQL metastore

• Type: Bug
• Status: Open
• Priority: Blocker
• Resolution: Unresolved
• Affects Version/s: 3.1.2
• Component/s: Metastore
• Assignee: wenjun ma
• Reporter: Dustin Koupal
• Created: 17/Jun/20 05:14
• Updated: 20/Jul/20 17:53

Description

When attempting to create a materialized view with transactions enabled, we get the following exception:

 

ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Failed to generate new Mapping of type org.datanucleus.store.rdbms.mapping.java.StringMapping, exception : JDBC type CLOB declared for field "org.apache.hadoop.hive.metastore.model.MCreationMetadata.txnList" of java type java.lang.String cant be mapped for this datastore.ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Failed to generate new Mapping of type org.datanucleus.store.rdbms.mapping.java.StringMapping, exception : JDBC type CLOB declared for field "org.apache.hadoop.hive.metastore.model.MCreationMetadata.txnList" of java type java.lang.String cant be mapped for this datastore.JDBC type CLOB declared for field "org.apache.hadoop.hive.metastore.model.MCreationMetadata.txnList" of java type java.lang.String cant be mapped for this datastore.org.datanucleus.exceptions.NucleusException: JDBC type CLOB declared for field "org.apache.hadoop.hive.metastore.model.MCreationMetadata.txnList" of java type java.lang.String cant be mapped for this datastore. at org.datanucleus.store.rdbms.mapping.RDBMSMappingManager.getDatastoreMappingClass(RDBMSMappingManager.java:1386) at org.datanucleus.store.rdbms.mapping.RDBMSMappingManager.createDatastoreMapping(RDBMSMappingManager.java:1616) at org.datanucleus.store.rdbms.mapping.java.SingleFieldMapping.prepareDatastoreMapping(SingleFieldMapping.java:59) at org.datanucleus.store.rdbms.mapping.java.SingleFieldMapping.initialize(SingleFieldMapping.java:48) at org.datanucleus.store.rdbms.mapping.RDBMSMappingManager.getMapping(RDBMSMappingManager.java:482) at org.datanucleus.store.rdbms.table.ClassTable.manageMembers(ClassTable.java:536) at org.datanucleus.store.rdbms.table.ClassTable.manageClass(ClassTable.java:442) at org.datanucleus.store.rdbms.table.ClassTable.initializeForClass(ClassTable.java:1270) at org.datanucleus.store.rdbms.table.ClassTable.initialize(ClassTable.java:276) at org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.initializeClassTables(RDBMSStoreManager.java:3279) at org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.run(RDBMSStoreManager.java:2889) at org.datanucleus.store.rdbms.AbstractSchemaTransaction.execute(AbstractSchemaTransaction.java:119) at org.datanucleus.store.rdbms.RDBMSStoreManager.manageClasses(RDBMSStoreManager.java:1627) at org.datanucleus.store.rdbms.RDBMSStoreManager.getDatastoreClass(RDBMSStoreManager.java:672) at org.datanucleus.store.rdbms.RDBMSStoreManager.getPropertiesForGenerator(RDBMSStoreManager.java:2088) at org.datanucleus.store.AbstractStoreManager.getStrategyValue(AbstractStoreManager.java:1271) at org.datanucleus.ExecutionContextImpl.newObjectId(ExecutionContextImpl.java:3760) at org.datanucleus.state.StateManagerImpl.setIdentity(StateManagerImpl.java:2267) at org.datanucleus.state.StateManagerImpl.initialiseForPersistentNew(StateManagerImpl.java:484) at org.datanucleus.state.StateManagerImpl.initialiseForPersistentNew(StateManagerImpl.java:120) at org.datanucleus.state.ObjectProviderFactoryImpl.newForPersistentNew(ObjectProviderFactoryImpl.java:218) at org.datanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionContextImpl.java:2079) at org.datanucleus.ExecutionContextImpl.persistObjectWork(ExecutionContextImpl.java:1923) at org.datanucleus.ExecutionContextImpl.persistObject(ExecutionContextImpl.java:1778) at org.datanucleus.ExecutionContextThreadedImpl.persistObject(ExecutionContextThreadedImpl.java:217) at org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:724) at org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:749) at org.apache.hadoop.hive.metastore.ObjectStore.createTable(ObjectStore.java:1308) at sun.reflect.GeneratedMethodAccessor54.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97) at com.sun.proxy.$Proxy25.createTable(Unknown Source) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1882) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1786) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:2035) at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108) at com.sun.proxy.$Proxy33.create_table_with_environment_context(Unknown Source) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14800) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14784) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

 

Steps to duplicate:

set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
set hive.compactor.worker.threads=1;
set hive.compactor.initiator.on=true;
set hive.exec.dynamic.partition.mode=nonstrict;
set hive.support.concurrency=true;
set hive.enforce.bucketing=true;

CREATE TABLE test (a STRING) STORED AS ORC TBLPROPERTIES ('transactional'='true');
CREATE MATERIALIZED VIEW test_foo AS SELECT * FROM test WHERE a = 'foo';