[IMPALA-13245] Table properties in HMS and Iceberg metadata do not align

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Frontend, Test
• Labels: impala-iceberg
• Epic Color: ghx-label-1
• Assignee: Unassigned
• Reporter: Noemi Pap-Takacs
• Created: 19/Jul/24 15:59
• Updated: 24/Sep/24 10:28

Description

Some Iceberg test tables in 'functional_parquet' used in Planner tests are missing some Iceberg table properties in the Iceberg metadata queried through IcebergApiTable, even though the properties are set in HMS (they show up in the impala-shell after issuing a 'DESCRIBE FORMATTED' command). Examples include: 'write.delete.mode', 'write.update.mode', 'write.merge.mode'
This causes different behavior depending on where we get the metadata in the code.

I tried to reproduce the issue creating Iceberg tables with Impala but most table properties were the same in HMS and Iceberg. E.g. 'write.delete.mode', etc. always defaulted to 'merge-on-read' showing up both in HMS and the Iceberg metadata. However, the test tables used in Planner tests were created with data-load, which could be an explanation.

Another issue is that when we alter a V1 to V2 by setting 'format-version'='2', the table does become a V2 table, but the 'format-version' property is not displayed in HMS.