[HIVE-27984] Support backward compatibility of hms thrift struct about column stats - ASF JIRA

Log work

Agile Board

Rank to Top

Rank to Bottom

Bulk Copy Attachments

Bulk Move Attachments

Voters

Watch issue

Watchers

Create sub-task

Convert to sub-task

Move

Link

Clone

Labels

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 4.0.0
Component/s: Standalone Metastore
Labels:
- pull-request-available

Description

We had some discussions about column stats thrift api isssue before: https://www.mail-archive.com/dev@hive.apache.org/msg144294.html ,

and i also did some attempt to fix this issue from TrinoDB side when i playing TrinoDB&Hive4 https://github.com/trinodb/trino-hive-apache/pull/43 .

Recently, i tried to use starrocks(https://github.com/StarRocks/starrocks) which uses a hms client compatible with Hive3 to query Hive4, and i also encountered this issue(https://github.com/StarRocks/starrocks/issues/38620),

Caused by: org.apache.thrift.TApplicationException: Required field 'engine' is unset! Struct:TableStatsRequest(dbName:testdb, tblName:testsr, colNames:[id], engine:null)
        at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) ~[libthrift-0.13.0.jar:0.13.0]
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_statistics_req(ThriftHiveMetastore.java:4161) ~[hive-apache-3.1.2-13.jar:?]
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_statistics_req(ThriftHiveMetastore.java:4148) ~[hive-apache-3.1.2-13.jar:?]
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTableColumnStatistics(HiveMetaStoreClient.java:735) ~[starrocks-fe.jar:?]
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTableColumnStatistics(HiveMetaStoreClient.java:727) ~[starrocks-fe.jar:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_221]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_221]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_221]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_221]
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:208) ~[hive-apache-3.1.2-13.jar:?]
        at com.sun.proxy.$Proxy32.getTableColumnStatistics(Unknown Source) ~[?:?]

In addition, i also found similar ticket in Impala https://issues.apache.org/jira/browse/IMPALA-12539 .

That is to say, if other components(Impala&TrinoDB&Starrocks) want to use hive3 hms client to get the column stats to make CBO, they will get the execption from Hive4. I think it is necessary to support backward compatibility to make other compontents work well with hive4(hms4).