[IMPALA-10254] Load data files via Iceberg for Iceberg Tables

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Component/s: Catalog
• Labels: impala-iceberg
• Epic Link: Iceberg support in Impala
• Epic Color: ghx-label-1
• Assignee: Tamas Mate
• Reporter: Zoltán Borók-Nagy
• Created: 19/Oct/20 08:56
• Updated: 29/Mar/23 09:53
• Resolved: 29/Mar/23 09:53

Description

Currently we still load the file descriptors of an Iceberg table via recursive file listing.

This lists too many files, e.g. metadata files, files that are being written (can later throw checksum errors), files from aborted INSERTs, removed files, etc.

We should use the Iceberg API to load the file descriptors corresponding to the table snapshot. Iceberg DataFiles might also already contain the split offsets.

Sub-Tasks

• Re-use FileDescriptors loaded by HdfsTable during IcebergTable load: Resolved