[IMPALA-11822] Optimize the Refresh/Invalidate event processing by skipping unnecessary events

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: Impala 4.3.0
• Component/s: Catalog
• Epic Link: event-processor-perf
• Epic Color: ghx-label-3
• Assignee: Sai Hemanth Gantasala
• Reporter: Sai Hemanth Gantasala
• Created: 04/Jan/23 01:34
• Updated: 04/Mar/24 08:28
• Resolved: 23/Mar/23 15:09

Description

Optimize the Refresh/Invalidate event processing by skipping unnecessary events.

Currently, we process every event as a new event. Consider there are 5 refresh events of the same table in the event processor queue. We can process the first refresh event and skip the remaining 4 events by comparing the timestamp of the refreshed table and event time. This way we can greatly improve the performance by skipping unnecessary events.

Issue Links

• causes: IMPALA-12855 NullPointerException in firing RELOAD events if the partition is just dropped
• causes: IMPALA-12213 Not a self-event because service id of this catalog does not match with one in event
• is related to: IMPALA-12865 enable_reload_events breaks enable_skipping_older_events by pushing lastRefreshEventId too high