[IMPALA-10949] Improve batching logic of events

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 4.4.0
• Component/s: Catalog
• Epic Link: event-processor-perf
• Epic Color: ghx-label-8
• Assignee: Sai Hemanth Gantasala
• Reporter: Vihang Karajgaonkar
• Created: 01/Oct/21 17:36
• Updated: 13/Feb/24 23:40
• Resolved: 15/Dec/23 16:58

Description

This is a followup based on the review comment https://gerrit.cloudera.org/#/c/17848/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1641

Current approach of batching batches together the events from a single operation so that self-event check is done per-batch. However, it looks like there is a considerable scope of improving the batching logic by clubbing together accross the various sources of the events on a table when IMPALA-10926 is merged. After IMPALA-10926 each table will track the last_synced_event and then the events processor can simply ignore a event which <= the last_synced_event. This simplification of self-events logic will enable easier batching for all the events of a type on a table.

Issue Links

• Dependent: IMPALA-10926 Sync db/table to latest event id for ddl operations in catalog metastore server
• is related to: IMPALA-12678 test_skipping_batching_events intermittent failure
• relates to: IMPALA-9857 Batch ALTER_PARTITION events