[IMPALA-11388] Add support for equality-based deletes

• Type: Epic
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Frontend
• Labels: impala-iceberg
• Epic Color: ghx-label-14
• Assignee: Gabor Kaszab
• Reporter: Zoltán Borók-Nagy
• Created: 23/Jun/22 13:01
• Updated: 19/Dec/23 22:42

Description

Iceberg V2 adds support for row-level modifications.
One way to implement this is via equality based delete files:
https://iceberg.apache.org/spec/#equality-delete-files
https://iceberg.apache.org/spec/#scan-planning

We could implement this via doing ANTI HASH JOIN between data and delete files. Similarly to what we do for Hive full ACID tables:
https://github.com/apache/impala/blob/f5fc08573352d0a1943296209791a4db17268086/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java#L1729-L1735

The complexity comes when different delete files use different set of columns. In that case we will need multiple ANTI HASH JOINs on top of each other.

Sub-Tasks

• Support equality delete files that don't contain the partition values: Open
• Push down conjuncts to the equality delete scanner: Open
• Missing field ID in the eq-delete file could filter out rows with null values: Open
• Use max(data_sequence_number) fo joining equality delete rows: Open
• Add better cardinality estimation for Iceberg V2 tables with equality deletes: Open