[IMPALA-11534] Skip reloading file metadata for some ALTER_TABLE events

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 4.3.0
• Fix Version/s: Impala 4.3.0
• Component/s: Catalog
• Epic Link: event-processor-perf
• Epic Color: ghx-label-13
• Assignee: Sai Hemanth Gantasala
• Reporter: Quanlong Huang
• Created: 25/Aug/22 02:22
• Updated: 27/Mar/24 14:02
• Resolved: 26/Jun/23 17:42

Description

EventProcessor will ignore trivial ALTER_TABLE events that only modify tblproperties like "transient_lastDdlTime", "totalSize", "numFilesErasureCoded", and "numFiles" (IMPALA-8579). For other kinds of non-rename ALTER_TABLE events, it will trigger a full refresh on the table, which is expensive when there are tens of thousands of partitions.

We can actually skip reloading file metadata for some ALTER_TABLE events, e.g. events that

• changing table comment
• adding/dropping columns
• changing column definition (name/type/comment)
• changing ownership
• setting customized tblproperties

Issue Links

• causes: IMPALA-12835 Transactional tables are unsynced when hms_event_incremental_refresh_transactional_table is disabled
• relates to: IMPALA-8579 Ignore trivial alter events
• relates to: IMPALA-11050 Skip file metadata reloading in AlterPartition event from event processor in catalogd