[HIVE-27984] Support backward compatibility of hms thrift struct about column stats

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0
• Component/s: Standalone Metastore
• Labels: pull-request-available
• Assignee: Butao Zhang
• Reporter: Butao Zhang
• Created: 08/Jan/24 02:50
• Updated: 16/Sep/24 16:31
• Resolved: 01/Feb/24 06:29

Description

We had some discussions about column stats thrift api isssue before: https://www.mail-archive.com/[e-mail]/msg144294.html ,

and i also did some attempt to fix this issue from TrinoDB side when i playing TrinoDB&Hive4 https://github.com/trinodb/trino-hive-apache/pull/43 .

Recently, i tried to use starrocks(https://github.com/StarRocks/starrocks) which uses a hms client compatible with Hive3 to query Hive4, and i also encountered this issue(https://github.com/StarRocks/starrocks/issues/38620), 

 

Caused by: org.apache.thrift.TApplicationException: Required field 'engine' is unset! Struct:TableStatsRequest(dbName:testdb, tblName:testsr, colNames:[id], engine:null)
        at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) ~[libthrift-0.13.0.jar:0.13.0]
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_statistics_req(ThriftHiveMetastore.java:4161) ~[hive-apache-3.1.2-13.jar:?]
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_statistics_req(ThriftHiveMetastore.java:4148) ~[hive-apache-3.1.2-13.jar:?]
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTableColumnStatistics(HiveMetaStoreClient.java:735) ~[starrocks-fe.jar:?]
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTableColumnStatistics(HiveMetaStoreClient.java:727) ~[starrocks-fe.jar:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_221]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_221]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_221]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_221]
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:208) ~[hive-apache-3.1.2-13.jar:?]
        at com.sun.proxy.$Proxy32.getTableColumnStatistics(Unknown Source) ~[?:?]
 

 

In addition, i also found similar ticket in Impala https://issues.apache.org/jira/browse/IMPALA-12539 .

That is to say, if other components(Impala&TrinoDB&Starrocks) want to use hive3 hms client to get the column stats to make CBO, they will get the execption from Hive4. I think it is necessary to support  backward compatibility to make other compontents work well with hive4(hms4).

Issue Links

• causes: IMPALA-13386 Fix test_metastore_service.py after HIVE-27984
• is related to: IMPALA-12539 hive-4 compatibility with Impala
• relates to: HIVE-22046 Differentiate among column stats computed by different engines
• links to: GitHub Pull Request #4984