[IMPALA-12895] REFRESH doesn't detect changes in partition locations in ACID tables

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Catalog
• Epic Color: ghx-label-11
• Assignee: Unassigned
• Reporter: Csaba Ringhofer
• Created: 12/Mar/24 15:20
• Updated: 12/Mar/24 15:21

Description

This was discovered by running test metadata.test_event_processing.TestEventProcessing.test_transact_partition_location_change_from_hive when flag hms_event_incremental_refresh_transactional_table  is set to false.

https://github.com/apache/impala/blob/ab6c9467f6347671b971dbce4c640bea032b6ed9/tests/metadata/test_event_processing.py#L164

 

When hms_event_incremental_refresh_transactional_table  is true (default), the alter partition event is processed correctly and the location change is detected. But if it is false or event processing is turned off, the change is not detected and running REFRESH on the table also doesn't update the location.

The different handling based on the flag seems intentional:

https://github.com/apache/impala/blob/ab6c9467f6347671b971dbce4c640bea032b6ed9/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#L2606

 

This seems to be an old issues while the test was added in a recent commit:

https://github.com/apache/impala/commit/32b29ff36fb3e05fd620a6714de88805052d0117

 

Issue Links

• relates to: IMPALA-12835 Transactional tables are unsynced when hms_event_incremental_refresh_transactional_table is disabled