[IMPALA-13240] Add tools to detect incompatible changes in catalog and statestore service

• Type: Task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: Impala 4.5.0
• Component/s: Infrastructure
• Epic Color: ghx-label-5
• Assignee: Quanlong Huang
• Reporter: Quanlong Huang
• Created: 18/Jul/24 03:08
• Updated: 4 days ago 11:28
• Resolved: 16/Aug/24 07:46

Description

To support upgrading only the impalads of a cluster, we need to make sure backward compatibility between impalad and the catalogd and statestore services. If there are breaking changes in Thrift/Flatbuffers definitions used in these services, the versions become incompatible. Note that Protobuf is only used in communication between impalads so can be ignored here.

The following files need to be checked:

• common/thrift/*.thrift
• common/fbs/*.fbs

Ideally changes in Thrift definitions should only add optional fields. Changes in Flatbuffers schema only add new fields at the end of a table definition.

We need a script to verify all Gerrit changes. A precommit job running this script and post comments will be helpful.

Ref:
https://diwakergupta.github.io/thrift-missing-guide/#_defining_structs
https://flatbuffers.dev/flatbuffers_guide_writing_schema.html

Issue Links

• is related to: HBASE-25340 Protobuf Mesage Incompatibility Detector
• is related to: IMPALA-10370 Thrift Message Incompatibility Detector
• is related to: IMPALA-13305 Better Thrift/FlatBuffers compatibility checks based on AST parsing