[IMPALA-11662] Improve "refresh iceberg_tbl_on_oss;" performance

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Resolved
• Fix Version/s: Impala 4.3.0
• Labels: impala-iceberg
• Epic Color: ghx-label-9
• Assignee: Li Penglin
• Reporter: Li Penglin
• Created: 14/Oct/22 03:56
• Updated: 13/Feb/23 17:55
• Resolved: 09/Feb/23 10:58

Description

Since Iceberg provides rich metadata, the cost of directory listing on OSS service e.g. S3A is higher than the cost on HDFS, we could create the file descriptors from Iceberg metadata instead of using org.apache.hadoop.fs.FileSystem#listFiles. https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java#L189.
The only thing missing there is the last_modification_time of the files. But since Iceberg files are immutable, maybe we could just come up with a special timestamp for these files.