[IMPALA-13116] In local-catalog mode, abort REFRESH and metadata reloading of DDL/DMLs if the table is invalidated

• Type: Improvement
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Component/s: Catalog
• Labels: catalog-2024
• Epic Color: ghx-label-8
• Assignee: Quanlong Huang
• Reporter: Quanlong Huang
• Created: 30/May/24 13:27
• Updated: 30/May/24 18:12

Description

A table can be invalidated when there are DDL/DML/REFRESHs running in flight:

• User can explictly trigger an INVALIDATE METADATA <table> command
• The table could be invalidated by CatalogdTableInvalidator when invalidate_tables_on_memory_pressure or invalidate_tables_timeout_s is turned on

Note that invalidating a table doesn't require holding the lock of the HdfsTable object so it can finish even if there are on-going updates on the table.

The updated HdfsTable object won't be added to the metadata cache since it has been replaced with an IncompleteTable object. It's only used in the DDL/DML/REFRESH responses. In local catalog mode, the response is the minimal representation which is mostly the table name and catalog version. We don't need the updates on the HdfsTable object to be finished. Thus, we can consider aborting the reloading of such DDL/DML/REFRESH requests.