[IMPALA-12835] Transactional tables are unsynced when hms_event_incremental_refresh_transactional_table is disabled

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: Impala 4.4.0
• Component/s: Catalog
• Epic Link: event-processor-completeness
• Epic Color: ghx-label-9
• Assignee: Csaba Ringhofer
• Reporter: Quanlong Huang
• Created: 22/Feb/24 12:02
• Updated: 30/Apr/24 07:06
• Resolved: 30/Apr/24 07:06

Description

There are some test failures when hms_event_incremental_refresh_transactional_table is disabled:

• tests/metadata/test_event_processing.py::TestEventProcessing::test_transactional_insert_events
• tests/metadata/test_event_processing.py::TestEventProcessing::test_event_based_replication

I can reproduce the issue locally:

$ bin/start-impala-cluster.py --catalogd_args=--hms_event_incremental_refresh_transactional_table=false
impala-shell> create table txn_tbl (id int, val int) stored as parquet tblproperties ('transactional'='true','transactional_properties'='insert_only');
impala-shell> describe txn_tbl;  -- make the table loaded in Impala

hive> insert into txn_tbl values(101, 200);

impala-shell> select * from txn_tbl; 

Impala shows no results until a REFRESH runs on this table.

Issue Links

• is caused by: IMPALA-11534 Skip reloading file metadata for some ALTER_TABLE events
• is related to: IMPALA-12895 REFRESH doesn't detect changes in partition locations in ACID tables
• is related to: IMPALA-12902 Event replication can be broken if hms_event_incremental_refresh_transactional_table=false
• relates to: IMPALA-10923 Fine grained table refreshing at partition level events for transactional tables