[IMPALA-11784] Don't call Iceberg's planFiles redundantly during table load

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 4.3.0
• Component/s: Catalog
• Labels: impala-iceberg
• Epic Color: ghx-label-9
• Assignee: Zoltán Borók-Nagy
• Reporter: Zoltán Borók-Nagy
• Created: 09/Dec/22 14:07
• Updated: 16/Aug/23 19:40
• Resolved: 13/Dec/22 12:21

Description

Iceberg's planFiles() API is very expensive because it involves reading the Avro manifest files. It's especially expensive on object stores, though manifest caching can help here.

Currently we invoke this API two times during table loading (via IcebergUtil.getIcebergFiles()), once in loadAllPartition() and once in loadPartitionStats().

We should just invoke IcebergUtil.getIcebergFiles() once, then pass the result object to loadAllPartition() and loadPartitionStats().

Issue Links

• is related to: IMPALA-11658 Implement Iceberg manifest caching configuration for Impala