[IMPALA-11032] Automatic Refresh of Metadata for Local Catalog after Compaction

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 4.1.0
• Component/s: Catalog
• Epic Color: ghx-label-4
• Assignee: Yu-Wen Lai
• Reporter: Yu-Wen Lai
• Created: 19/Nov/21 17:24
• Updated: 22/Dec/21 05:13
• Resolved: 22/Dec/21 05:13

Description

After Hive compaction for tables created in the Hive warehouse, queries(HIVE ACID table) made in Impala possibly fail with a FileNotFoundException if a file removed by the Hive cleaner.

In IMPALA-10801, we check latest compaction id before serving metadata from Catalogd. However, coordinators don't take advantage of that. Coordinators have local cache so we will have do the same compaction check for coordinator as well.

Issue Links

• Dependent: IMPALA-11045 Should start transaction when auto_check_compaction is enabled