[IMPALA-7954] Support automatic invalidates using metastore notification events

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 3.1.0
• Fix Version/s: Impala 4.1.0
• Component/s: Catalog
• Assignee: Vihang Karajgaonkar
• Reporter: Vihang Karajgaonkar
• Created: 10/Dec/18 22:44
• Updated: 29/Dec/22 02:57
• Resolved: 09/Aug/21 17:35

Description

Currently, in Impala there are multiple ways to invalidate or refresh the metadata stored in Catalog for Tables. Objects in Catalog can be invalidated either on usage based approach (invalidate_tables_timeout_s) or when there is GC pressure (invalidate_tables_on_memory_pressure) as added in IMPALA-7448. However, most users issue invalidate commands when they want to sync to the latest information from HDFS or HMS. Unfortunately, when data is modified or new data is added outside Impala (eg. Hive) or a different Impala cluster, users don't have a clear idea on whether they have to issue invalidate or not. To be on the safer side, users keep issuing invalidate commands more than necessary and it causes performance as well as stability issues.

Hive Metastore provides a simple API to get incremental updates to the metadata information stored in its database. Each API which does a add/alter/drop operation in metastore generates event(s) which can be fetched using get_next_notification API. Each event has a unique and increasing event_id. The current notification event id can be fetched using get_current_notificationEventId API.

This JIRA proposes to make use of such events from metastore to proactively either invalidate or refresh information in the catalogD. When configured, CatalogD could poll for such events and take action (like add/drop/refresh partition, add/drop/invalidate tables and databases) based on the events. This way we can automatically refresh the catalogD state using events and it would greatly help the use-cases where users want to see the latest information (within a configurable interval of time delay) without flooding the system with invalidate requests.

I will be attaching a design doc to this JIRA and create subtasks for the work. Feel free to make comments on the JIRA or make suggestions to improve the design.

Issue Links

• causes: IMPALA-10468 DROP events which are generated while a batch is being processed may add table incorrectly
• incorporates: IMPALA-8592 Add support for insert events for 'LOAD DATA..' statements from Impala.
• incorporates: IMPALA-10273 Support function events
• incorporates: IMPALA-9857 Batch ALTER_PARTITION events
• is duplicated by: IMPALA-3124 Add external metadata notification mechanisms to Impala
• is related to: IMPALA-11533 EventProcessor Completeness
• is related to: IMPALA-9101 Unneccessary REFRESH due to wrong self-event detection
• is related to: IMPALA-5151 Adding partition on impala over a table with old metadata
• is related to: IMPALA-8600 Reload partition does not work for transactional tables
• is related to: IMPALA-10923 Fine grained table refreshing at partition level events for transactional tables
• relates to: IMPALA-4272 Automatic invalidation HDFS file and block metadata
• links to: Design_Doc

Sub-Tasks

• Add support for automatic invalidates by polling metastore events: Resolved
• Add support to detect insert events from Impala: Resolved
• Detect self-events to avoid unnecessary invalidates: Resolved
• Add support for fine-grained updates at partition level: Resolved
• Impala Doc: Doc the options to enable automatic invalidates using metastore notification events: Closed
• Improve supportability of the automatic invalidate feature: Resolved
• Add a flag to disable sync using events at a table level: Resolved
• Impala Doc: Doc the support for fine-grained updates at partition level: Closed
• Add support for alter_database events: Resolved
• Event processor should keep trying if metastore is unavailable: Resolved
• Event filtering logic may not filter all the events: Resolved
• Change metastore configuration template so that table parameters do not exclude impala specific properties: Resolved
• Fix MetastoreEventsProcessorTest flakiness: Resolved
• Impala Doc: Doc the flag to disable sync using events at the table level: Closed
• Impala Doc: Doc the supportability metrics of the automatic metadata invalidate feature: Closed
• Check CREATION_TIME of databases in event processor to avoid incorrect/redundant invalidates: Resolved
• Fetch metastore configuration values to detect misconfigured setups: Resolved
• Impala Doc: Document the feature to detect insert events from Impala: Closed
• Impala Doc: Doc the feature for alter_database events: Closed
• Support config validation for event processor on HMS-3: Resolved
• Ignore trivial alter events: Resolved
• Add support for self-event detection for insert events: Resolved
• Insert event should not error when table does not exists: Resolved
• Enable Event processing by default Impala e2e tests: Resolved
• Create randomized tests for stressing the event processor: Resolved
• Configuration validation introduced in IMPALA-8559 can be improved: Closed
• Enable event polling by default in tests: Resolved