[IMPALA-12712] INVALIDATE METADATA <table> should set a better createEventId

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: Impala 4.5.0, Impala 4.4.1
• Component/s: Catalog
• Labels: catalog-2024
• Epic Link: event-processor-completeness
• Epic Color: ghx-label-14
• Assignee: Sai Hemanth Gantasala
• Reporter: Quanlong Huang
• Created: 15/Jan/24 08:34
• Updated: 14/Aug/24 00:18
• Resolved: 22/Jul/24 19:06

Description

"INVALIDATE METADATA <table>" can be used to bring up a table in Impala's catalog cache if the table exists in HMS. For instance, when HMS event processing is disabled, we can use it in Impala to bring up tables that are created outside Impala.

The createEventId for such tables are always set as -1:
https://github.com/apache/impala/blob/6ddd69c605d4c594e33fdd39a2ca888538b4b8d7/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L2243-L2246

This is problematic when event-processing is enabled. DropTable events and RenameTable events use the createEventId to decide whether to remove the table in catalog cache. -1 will lead to always removing the table. Though it might be added back shortly in follow-up CreateTable events, in the period between them the table is missing in Impala, causing test failures like IMPALA-12266.

A simpler reproducing of the issue is creating a table in Hive and launching Impala with a long event polling interval to mimic the delay on events. Note that we start Impala cluster after creating the table so Impala don't need to process the CREATE_TABLE event.

hive> create table debug_tbl (i int);

bin/start-impala-cluster.py --catalogd_args=--hms_event_polling_interval_s=60

Drop the table in Impala and recreate it in Hive, so it doesn't exist in the catalog cache but exist in HMS. Run "INVALIDATE METADATA <table>" in Impala to bring it up before the DROP_TABLE event come.

impala> drop table debug_tbl;
hive> create table debug_tbl (i int, j int);
impala> invalidate metadata debug_tbl;

The table will be dropped by the DROP_TABLE event and then added back by the CREATE_TABLE event. Shown in catalogd logs:

I0115 16:30:15.376713  3208 JniUtil.java:177] 02457b6d5f174d1f:3bdeee1400000000] Finished execDdl request: DROP_TABLE default.debug_tbl issued by quanlong. Time spent: 417ms

I0115 16:30:23.390962  3208 CatalogServiceCatalog.java:2777] 1840bd101f78d611:22079a5a00000000] Invalidating table metadata: default.debug_tbl
I0115 16:30:23.404150  3208 Table.java:234] 1840bd101f78d611:22079a5a00000000] createEventId_ for table: default.debug_tbl set to: -1
I0115 16:30:23.405138  3208 JniUtil.java:177] 1840bd101f78d611:22079a5a00000000] Finished resetMetadata request: INVALIDATE TABLE default.debug_tbl issued by quanlong. Time spent: 17ms

I0115 16:30:55.108006 32760 MetastoreEvents.java:637] EventId: 8668853 EventType: DROP_TABLE Successfully removed table default.debug_tbl

I0115 16:30:55.108459 32760 MetastoreEvents.java:637] EventId: 8668855 EventType: CREATE_TABLE Successfully added table default.debug_tbl

CC VenuReddy, hemanth619

Issue Links

• causes: IMPALA-12266 Sporadic failure after migrating a table to Iceberg