[IMPALA-10923] Fine grained table refreshing at partition level events for transactional tables

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 4.1.0
• Component/s: Catalog
• Epic Link: event-processor-perf
• Target Version: Impala 4.1.0
• Epic Color: ghx-label-9
• Assignee: Yu-Wen Lai
• Reporter: Yu-Wen Lai
• Created: 20/Sep/21 18:16
• Updated: 22/Feb/24 12:02
• Resolved: 18/Feb/22 02:08

Description

For ensuring the transactional tables are consistent, we currently take whole table refreshing even a change is just for a partition only. That is too expensive and possibly make event processing has a longer delay.

To enable fine-grained table refreshing, there are three main changes in this proposal.

1. maintain validWriteIdList in Catalogd for transactional tables. We will track write id changes by AllocWriteIdEvents, CommitTxnEvents, and AbortTxnEvents.
2. trigger partition level refreshing for addPartitionEvents, dropPartitionEvents, and AlterPartitionEvents.
3. Introduce a config incremental_refresh_acid, which can switch on/off the fine-grained table refreshing

Issue Links

• causes: IMPALA-11093 Fine grained table refreshing doesn't refresh table file metadata
• is related to: IMPALA-12835 Transactional tables are unsynced when hms_event_incremental_refresh_transactional_table is disabled
• relates to: IMPALA-7954 Support automatic invalidates using metastore notification events