[IMPALA-10370] Thrift Message Incompatibility Detector

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Infrastructure
• Epic Color: ghx-label-11
• Assignee: Unassigned
• Reporter: junwen yang
• Created: 01/Dec/20 17:59
• Updated: 22/Mar/21 15:36

Description

Regarding the issue  IMPALA-8243 caused by the incompatibility of thrift message, we have created a static checker which keeps track of the thrift file change, and detect potential incompatibility:

1. Add/delete required field.  The thrift guidelines suggests Any new fields that you add should be optional.
2. The tag number of a field has been changed. Also, the thrift guidelines suggests Don’t change the numeric tags for any existing fields.
3. A  required field has been changed to optional, or an optional field has been changed to required. According to the guidelines , Required Is Forever You should be very careful about marking fields as required. If at some point you wish to stop writing or sending a required field, it will be problematic to change the field to an optional field — old readers will consider messages without this field to be incomplete and may reject or drop them unintentionally. You should consider writing application-specific custom validation routines for your buffers instead. Some have come to the conclusion that using required does more harm than good; they prefer to use only optional. However, this view is not universal.

We have applied our checker on the frequently maintained IMPALA versions: refs/tags/2.10.0, refs/tags/2.11.0, refs/tags/2.12.0, refs/tags/2.7.0, refs/tags/2.8.0, refs/tags/2.9.0, refs/tags/3.0.0, refs/tags/3.0.1, refs/tags/3.1.0, refs/tags/3.2.0, refs/tags/3.3.0, refs/tags/3.4.0, we found more than 1000 problems as attached. 

The results reported by our checker got confirmed by developers of HBASE and our checker is requested by them, which can be found at HBASE-25340.