[IMPALA-12865] enable_reload_events breaks enable_skipping_older_events by pushing lastRefreshEventId too high

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Component/s: Catalog
• Epic Link: event-processor-perf
• Epic Color: ghx-label-4
• Assignee: Sai Hemanth Gantasala
• Reporter: Quanlong Huang
• Created: 04/Mar/24 08:20
• Updated: 04/Mar/24 08:28

Description

When enable_reload_events is true, catalogd fires RELOAD events after REFRESH finishes reloading the table/partition. The RELOAD event id is also used to update lastRefreshEventId of the table/partition.
https://github.com/apache/impala/blob/784971c018c2dc44c53d7c0f366ad49cd8681ac6/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L6807-L6814

This is problematic when enable_skipping_older_events is true. HMS events generated after the reload and before the RELOAD event will be skipped.
https://github.com/apache/impala/blob/784971c018c2dc44c53d7c0f366ad49cd8681ac6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#L1219-L1221

The issue is identified in the review of https://gerrit.cloudera.org/c/21096/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#6799

CC hemanth619, VenuReddy, csringhofer

Issue Links

• relates to: IMPALA-11535 Skip events happen before manual REFRESH
• relates to: IMPALA-11822 Optimize the Refresh/Invalidate event processing by skipping unnecessary events