[IMPALA-10453] Support file/partition pruning via runtime filters on Iceberg

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Resolved
• Fix Version/s: Impala 4.2.0
• Component/s: Backend
• Labels: iceberg impala-iceberg performance
• Epic Link: Iceberg support in Impala
• Epic Color: ghx-label-5
• Assignee: Tamas Mate
• Reporter: Tim Armstrong
• Created: 25/Jan/21 18:54
• Updated: 25/Oct/22 09:28
• Resolved: 11/Aug/22 11:40

Description

This is a placeholder to figure out what we'd need to do to support dynamic file-level pruning in Iceberg using runtime filters, i.e. have parity for partition pruning.

• If there is a single partition value per file, then applying bloom filters to the row group stats would be effective at pruning files.
• If there are partition transforms, e.g. hash-based, then I think we probably need to track the partition that the file is associated with and then have some custom logic in the parquet scanner to do partition pruning.