[IMPALA-9937] Catalogd should send incremental metadata updates in DDL responses to legacy coordinators

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Catalog
• Epic Link: fine-grained metadata in catalogd
• Epic Color: ghx-label-5
• Assignee: Unassigned
• Reporter: Quanlong Huang
• Created: 09/Jul/20 08:13
• Updated: 24/Sep/24 09:56

Description

After IMPALA-3127, catalogd is able to send incremental metadata updates in partition level. Ideally, legacy coordinators should receive incremental updates in DDL responses. E.g. when adding one partition, only get back the metadata of this partition and some table level updates, not including metadata of other unchanged partitions.

However, catalogd still sends full table objects in DDL responses because catalogd is not aware of the table metadata cached in coordinator's cache. Catalogd can't compare them to send incremental updates. To support this, the Catalogd RPC requests should include the partition id list of the target table, catalogd uses it to generate incremental metadata updates in responses.

Issue Links

• is related to: IMPALA-13385 The response of ddl exceeds VM limit when serializing the catalog
• relates to: IMPALA-9936 Only send invalidations in DDL responses to LocalCatalog coordinators