[IMPALA-11808] Improve catalogD cache performance in a multi service environment for a refresh/invalidate query

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 4.3.0
• Component/s: Catalog
• Epic Link: event-processor-perf
• Epic Color: ghx-label-13
• Assignee: Sai Hemanth Gantasala
• Reporter: Sai Hemanth Gantasala
• Created: 19/Dec/22 17:07
• Updated: 28/Mar/24 20:03
• Resolved: 26/Jan/23 19:47

Description

Improve catalogD cache performance in a multi-service environment for a refresh/invalidate query.

This patch supports a new event called reload event in catalogD. This event is used to update table/file metadata for a refresh or invalidate command in other replicas of catalogDs when one of the replicas issues a refresh/invalidate command. HIVE-26838 is the Hive jira that adds support for this reload event. This feature is disabled by default using a config enable_reload_events. To use this feature set this config to true and impala will be able to fire reload events. The processing of reload events in the event processor is always enabled. There is an end-to-end test added for this feature which currently checks the firing/creation of the reload event and self-event check in the event processor. There is also an end-to-end test that tests this reload event in the event processor.

There is also a follow-up jira IMPALA-11822 to track the optimization patch for this feature.

Issue Links

• causes: IMPALA-12082 INVALIDATE METADATA on new table fail if db is not loaded
• requires: HIVE-26838 Add a new event to improve cache performance in external systems that communicates with HMS.