[IMPALA-12327] Iceberg V2 operator wrong results in PARTITIONED mode

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 4.3.0
• Component/s: Backend
• Labels: impala-iceberg
• Epic Color: ghx-label-3
• Assignee: Zoltán Borók-Nagy
• Reporter: Zoltán Borók-Nagy
• Created: 01/Aug/23 12:56
• Updated: 26/Mar/24 21:25
• Resolved: 03/Aug/23 15:25

Description

The Iceberg delete node tries to do mini merge-joins between data records and delete records. This works in DISTRIBUTED mode, and most of the time in PARTITIONED mode as well. The Iceberg delete node had the wrong assumption that if the rows in a row batch belong to the same file, and come in ascending order, we don't need to update the IcebergDeleteState which tracks the state of the probing.

But when PARTITIONED mode is used, we cannot rely on ascending row order, not even inside row batches, not even when the previous file path is the same as the current one.

This is because files with multiple blocks can be processed by multiple hosts in parallel, then the rows are getting hash-exchanged based on their file paths. Then the exchange-receiver at the LHS coalesces the row batches from multiple senders, hence the row IDs getting unordered.

Issue Links

• is caused by: IMPALA-11619 Improve Iceberg V2 reads with a custom Iceberg Position Delete operator