[HBASE-25340] Protobuf Mesage Incompatibility Detector

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.3.3, 2.2.6
• Assignee: Unassigned
• Reporter: junwen yang
• Created: 29/Nov/20 04:37
• Updated: 29/Jul/24 16:02

Description

Regarding the issue HBASE-25238. caused by the incompatibility of protobuf message, we have created a static checker which keeps track of the proto file change, and detect potential incompatibility:

1. Add/delete required field, which is exactly the case reported in HBASE-25238.
2. The tag number of a field has been changed, as described in HDFS-9788. Also, the protobuf guidelines suggests each field in the message definition has a unique number. These numbers are used to identify your fields in the message binary format, and should not be changed once your message type is in use. 
3. A  required field has been changed to optional, or an optional field has been changed to required. According to the guidelines in protobuf official website, Required Is Forever You should be very careful about marking fields as required. If at some point you wish to stop writing or sending a required field, it will be problematic to change the field to an optional field - old readers will consider messages without this field to be incomplete and may reject or drop them unintentionally. You should consider writing application-specific custom validation routines for your buffers instead.

We have applied our checker on the frequently maintained HBASE versions: rel/1.4.3, rel/1.6.0, rel/2.2.6, rel/2.3.3, and master, and we found 37 potential problems as attached, which includes the one found in HBASE-25238. 

 

Issue Links

• duplicates: HBASE-25344 Detector
• relates to: HBASE-25238 Upgrading HBase from 2.2.0 to 2.3.x fails because of “Message missing required fields: state”
• relates to: IMPALA-13240 Add tools to detect incompatible changes in catalog and statestore service