[HIVE-23145] get_partitions_with_specs fails if filter expression is not parsable

• Type: Sub-task
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: Metastore
• Assignee: Vineet Garg
• Reporter: Vineet Garg
• Created: 06/Apr/20 18:20
• Updated: 11/Apr/20 00:02

Description

Expression is not parsable in most of the cases. Current API get_partitions_by_expr anticipates this and provide a fallback mechanism. This basically deserialize the provided expression, fetches all partition names for the table, prune partition names using the expression and then uses the names to fetch required partition data.
Note that this expect serialized expression instead of string.

This need to be done for both Direct SQL and JDO path.

e.g. Following error is thrown for tpcds query 55 which provide expression <column> IS NOT NULL filter

ERROR

MetaException(message:Error parsing partition filter; lexer error: null; exception NoViableAltException(13@[]))MetaException(message:Error parsing partition filter; lexer error: null; exception NoViableAltException(13@[])) at org.apache.hadoop.hive.metastore.PartFilterExprUtil.getFilterParser(PartFilterExprUtil.java:154) at org.apache.hadoop.hive.metastore.ObjectStore$15.initExpressionTree(ObjectStore.java:4339) at org.apache.hadoop.hive.metastore.ObjectStore$15.canUseDirectSql(ObjectStore.java:4319) at org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.start(ObjectStore.java:4021) at org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:3985) at org.apache.hadoop.hive.metastore.ObjectStore.getPartitionSpecsByFilterAndProjection(ObjectStore.java:4395) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97) at com.sun.proxy.$Proxy26.getPartitionSpecsByFilterAndProjection(Unknown Source) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partitions_with_specs(HiveMetaStore.java:5356) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108) at com.sun.proxy.$Proxy27.get_partitions_with_specs(Unknown Source) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_with_specs.getResult(ThriftHiveMetastore.java:21620) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_with_specs.getResult(ThriftHiveMetastore.java:21604) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:643) at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:638) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876) at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:638) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)