[IMPALA-11053] Impala should be able to read migrated partitioned Iceberg tables

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 4.1.0
• Labels: impala-iceberg
• Epic Link: Iceberg support in Impala
• Epic Color: ghx-label-10
• Assignee: Zoltán Borók-Nagy
• Reporter: Zoltán Borók-Nagy
• Created: 08/Dec/21 16:19
• Updated: 09/Jun/22 14:00
• Resolved: 08/Mar/22 13:00

Description

When Hive (and probably other engines as well) converts a legacy Hive table to Iceberg it doesn't rewrite the data files.

It means that the data files don't have write ids, moreover they don't have the partition columns neither.

Currently Impala expects tha partition columns to be present in the data files, so it won't be able to read converted partitioned tables.

So we need to inject partition values from the Iceberg metadata, plus resolve columns correctly (position-based resolution needs an offset).

Issue Links

• is related to: IMPALA-11034 Resolve schema of old data files in migrated Iceberg tables