[HIVE-25073] Optimise HiveAlterHandler::alterPartitions

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Component/s: Metastore
• Assignee: Harshit Gupta
• Reporter: Rajesh Balamohan
• Created: 29/Apr/21 08:07
• Updated: 12/Apr/23 16:56
• Resolved: 12/Apr/23 16:56

Description

Table details are populated again and again for each partition, which can be avoided.

https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java#L5892

https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java#L808

Following stacktrace may be relevant for apache master as well.

	at org.datanucleus.store.query.Query.executeWithArray(Query.java:1744)
	at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:368)
	at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:255)
	at org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:2113)
	at org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:2152)
	at org.apache.hadoop.hive.metastore.ObjectStore.alterPartitionNoTxn(ObjectStore.java:4951)
	at org.apache.hadoop.hive.metastore.ObjectStore.alterPartitions(ObjectStore.java:5057)
	at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
	at com.sun.proxy.$Proxy27.alterPartitions(Unknown Source)
	at org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartitions(HiveAlterHandler.java:798)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions_with_environment_context(HiveMetaStore.java:5695)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions_req(HiveMetaStore.java:5647)
	at sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
	at com.sun.proxy.$Proxy28.alter_partitions_req(Unknown Source)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_partitions_req.getResult(ThriftHiveMetastore.java:18557)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_partitions_req.getResult(ThriftHiveMetastore.java:18541)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:643)
	at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:638)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)