[IMPALA-12694] Test equality delete support with data from NiFi

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Not Applicable
• Component/s: Backend, Frontend
• Labels: impala-iceberg
• Epic Color: ghx-label-5
• Assignee: Gabor Kaszab
• Reporter: Gabor Kaszab
• Created: 09/Jan/24 09:06
• Updated: 09/Apr/24 09:09
• Resolved: 09/Apr/24 09:09

Description

Iceberg equality delete support in Impala is a subset of what the Iceberg spec allows for equality deletes. Currently, we have sufficient implementation to use eq-deletes created by Flink. As a next step, let's examine if this implementation is sufficient for eq-deletes created by NiFi.

In theory, NiFi uses Flink's eq-delete implementation so Impala should be fine reading such data. However, at least some manual tests needed for verification, and if it turns out that there are some uncovered edge cases, we should fill these holes in the implementation (probably in separate jiras).