[IMPALA-6994] Avoid reloading a table's HMS data for file-only operations

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: Impala 2.12.0
• Component/s: Catalog
• Epic Color: ghx-label-4
• Assignee: Unassigned
• Reporter: Balazs Jeszenszky
• Created: 08/May/18 19:54
• Updated: 09/May/21 03:40

Description

Reloading file metadata for HDFS tables (e.g. as a final step in an 'insert') is done via
https://github.com/apache/impala/blob/branch-2.12.0/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L628
, which calls
https://github.com/apache/impala/blob/branch-2.12.0/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java#L1243

HdfsTable.load has no option to only load file metadata. HMS metadata will also be reloaded every time, which is an unnecessary overhead (and potential point of failure) when adding files to existing locations.

Issue Links

• is related to: IMPALA-9779 Unnecessarily reloading file metadata in some DDLs