[IMPALA-11484] Create SCAN plan for Iceberg V2 position delete tables

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 4.2.0
• Component/s: Frontend
• Labels: impala-iceberg
• Epic Link: Iceberg V2 format
• Epic Color: ghx-label-13
• Assignee: Zoltán Borók-Nagy
• Reporter: Zoltán Borók-Nagy
• Created: 08/Aug/22 14:10
• Updated: 19/Sep/22 11:18
• Resolved: 19/Sep/22 11:18

Description

Iceberg position delete files store the full URIs and and file positions of rows that are deleted. Therefore we can do an ANTI HASH JOIN between data files and delete files to retrieve only the active rows.

For the data file rows we need to get the virtual columns INPUT_FILE_NAME and FILE_POSITION, while in the delete files we need to retrieve the columns 'file_path' and 'pos': https://iceberg.apache.org/spec/#position-delete-files

Since the data files are in table schema, and the delete files are in a different schema, we need to create a virtual table for the delete files with the corresponding schema.

Iceberg tells us which delete files must be applied to which data files, i.e. if a data file doesn't have a corresponding delete file, the content can be just UNION'ed with the output of the ANTI HASH JOIN.

See more information in the design doc: https://docs.google.com/document/d/1WF_UOanQ61RUuQlM4LaiRWI0YXpPKZ2VEJ8gyJdDyoY/edit#heading=h.5gc49pcc2543

Sub-Tasks

• Create better cardinality estimates for Iceberg V2 tables with deletes: Resolved
• Rewrite queries with unrelative collection columns for Iceberg tables: Open