[IMPALA-13499] REFRESH on Iceberg tables can lead to data loss

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: Impala 4.4.1
• Component/s: Catalog
• Epic Color: ghx-label-8
• Assignee: Unassigned
• Reporter: Saulius Valatka
• Created: 30/Oct/24 12:45
• Updated: 30/Oct/24 14:38

Description

When running a REFRESH statement on an Iceberg table the catalog loads it from the Hive metastore and later performs an alter_table here. It does so without taking a Hive lock, meaning that if any external process commits to the table between load and alter, the newly committed "metadata_location" property will be overwritten with the previous value and effectively will result in data loss.

It should either take a Hive lock when doing this, or, if "iceberg.engine.hive.lock-enabled = false" use "alter_table_with_environmentContext" and set expected_parameter_key / expected_parameter_value to metadata_location / <previous version>.

Issue Links

• relates to: IMPALA-13484 Querying an Iceberg table with TIMESTAMP_LTZ can cause data loss