Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-19715 Consolidated and flexible API for fetching partition metadata from HMS
  3. HIVE-23148

get_partitions_with_specs fail for postgres with argument type mismatch exception

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 4.0.0
    • None
    • None
    • None

    Description

      MetaException(message:java.lang.IllegalArgumentException: Cannot invoke org.apache.hadoop.hive.metastore.api.StorageDescriptor.setNumBuckets on bean class 'class org.apache.hadoop.hive.metastore.api.StorageDescriptor' - argument type mismatch - had objects of type "java.lang.Long" but expected signature "int"
	at org.apache.commons.beanutils.PropertyUtilsBean.invokeMethod(PropertyUtilsBean.java:2196)
	at org.apache.commons.beanutils.PropertyUtilsBean.setSimpleProperty(PropertyUtilsBean.java:2109)
	at org.apache.commons.beanutils.PropertyUtilsBean.setNestedProperty(PropertyUtilsBean.java:1915)
	at org.apache.commons.beanutils.PropertyUtils.setNestedProperty(PropertyUtils.java:866)
	at org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.setNestedProperty(MetaStoreUtils.java:1938)
	at org.apache.hadoop.hive.metastore.PartitionProjectionEvaluator$1.setValue(PartitionProjectionEvaluator.java:412)
	at org.apache.hadoop.hive.metastore.PartitionProjectionEvaluator.traverseAndSetValues(PartitionProjectionEvaluator.java:501)
	at org.apache.hadoop.hive.metastore.PartitionProjectionEvaluator.traverseAndSetValues(PartitionProjectionEvaluator.java:505)
	at org.apache.hadoop.hive.metastore.PartitionProjectionEvaluator.setSingleValuedFields(PartitionProjectionEvaluator.java:392)
	at org.apache.hadoop.hive.metastore.PartitionProjectionEvaluator.getPartitionsUsingProjectionList(PartitionProjectionEvaluator.java:358)
	at org.apache.hadoop.hive.metastore.MetaStoreDirectSql$3.run(MetaStoreDirectSql.java:642)
	at org.apache.hadoop.hive.metastore.Batchable.runBatched(Batchable.java:73)
	at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsUsingProjectionAndFilterSpec(MetaStoreDirectSql.java:639)
	at org.apache.hadoop.hive.metastore.ObjectStore$15.getSqlResult(ObjectStore.java:4344)
	at org.apache.hadoop.hive.metastore.ObjectStore$15.getSqlResult(ObjectStore.java:4315)
	at org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:3989)
	at org.apache.hadoop.hive.metastore.ObjectStore.getPartitionSpecsByFilterAndProjection(ObjectStore.java:4410)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
	at com.sun.proxy.$Proxy26.getPartitionSpecsByFilterAndProjection(Unknown Source)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partitions_with_specs(HiveMetaStore.java:5356)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
	at com.sun.proxy.$Proxy27.get_partitions_with_specs(Unknown Source)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_with_specs.getResult(ThriftHiveMetastore.java:21620)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_with_specs.getResult(ThriftHiveMetastore.java:21604)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:643)
	at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:638)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
	at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:638)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: argument type mismatch
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.commons.beanutils.PropertyUtilsBean.invokeMethod(PropertyUtilsBean.java:2128)
	... 44 more
)
	at org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.setNestedProperty(MetaStoreUtils.java:1941)
	at org.apache.hadoop.hive.metastore.PartitionProjectionEvaluator$1.setValue(PartitionProjectionEvaluator.java:412)
	at org.apache.hadoop.hive.metastore.PartitionProjectionEvaluator.traverseAndSetValues(PartitionProjectionEvaluator.java:501)
	at org.apache.hadoop.hive.metastore.PartitionProjectionEvaluator.traverseAndSetValues(PartitionProjectionEvaluator.java:505)
	at org.apache.hadoop.hive.metastore.PartitionProjectionEvaluator.setSingleValuedFields(PartitionProjectionEvaluator.java:392)
	at org.apache.hadoop.hive.metastore.PartitionProjectionEvaluator.getPartitionsUsingProjectionList(PartitionProjectionEvaluator.java:358)
	at org.apache.hadoop.hive.metastore.MetaStoreDirectSql$3.run(MetaStoreDirectSql.java:642)
	at org.apache.hadoop.hive.metastore.Batchable.runBatched(Batchable.java:73)
	at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsUsingProjectionAndFilterSpec(MetaStoreDirectSql.java:639)
	at org.apache.hadoop.hive.metastore.ObjectStore$15.getSqlResult(ObjectStore.java:4344)
	at org.apache.hadoop.hive.metastore.ObjectStore$15.getSqlResult(ObjectStore.java:4315)
	at org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:3989)
	at org.apache.hadoop.hive.metastore.ObjectStore.getPartitionSpecsByFilterAndProjection(ObjectStore.java:4410)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
	at com.sun.proxy.$Proxy26.getPartitionSpecsByFilterAndProjection(Unknown Source)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partitions_with_specs(HiveMetaStore.java:5356)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
	at com.sun.proxy.$Proxy27.get_partitions_with_specs(Unknown Source)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_with_specs.getResult(ThriftHiveMetastore.java:21620)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_with_specs.getResult(ThriftHiveMetastore.java:21604)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:643)
	at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:638)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
	at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:638)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
	Suppressed: java.lang.NoSuchMethodError: javax.jdo.Query.close()V
		at org.apache.hadoop.hive.metastore.PartitionProjectionEvaluator.setSingleValuedFields(PartitionProjectionEvaluator.java:456)
		... 36 more

      This is because StorageDescriptor.setNumBuckets expect int but postgres schema return long (bigint). This needs to be explicitly checked for and cast to int.

      Attachments

        Activity

          People

            Unassigned Unassigned
            vgarg Vineet Garg
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: