[IMPALA-10265] Doc about enable_incremental_metadata_updates flag

• Type: Documentation
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Component/s: Docs
• Epic Color: ghx-label-8
• Assignee: Unassigned
• Reporter: Quanlong Huang
• Created: 20/Oct/20 01:40
• Updated: 22/Dec/20 19:19

Description

IMPALA-10113 adds a feature flag to turn off the incremental metadata update feature which defaults to be turned on. This flag decides how catalogd propagates metadata updates to the catalog topic.

If enable_incremental_metadata_updates is true, catalogd will send metadata updates in partition granularity. So a table that just has one partition changed will only have an update on that partition. This reduces the size of the metadata that needs to be sent from the catalogd.

If enable_incremental_metadata_updates is false, that's the legacy behavior. Catalogd will send metadata updates in table granularity. So a table that just has one partition changed will still have an update for the whole table object. Catalogd still sends the whole table thrift object to the catalog topic.

Note that this is a catalogd-only flag. Don't need to be set on impalads since impalad can process both incremental and non-incremental catalog updates.

Issue Links

• relates to: IMPALA-10113 Add feature flag for incremental metadata update