[IMPALA-10801] Check the latest compaction Id before serving request

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 4.1.0
• Component/s: Catalog
• Epic Color: ghx-label-12
• Assignee: Yu-Wen Lai
• Reporter: Yu-Wen Lai
• Created: 15/Jul/21 22:20
• Updated: 03/Dec/21 05:39
• Resolved: 19/Aug/21 17:24

Description

Cache compaction Id for a given table/file-metadata in CatalogD.

Whenever there is a read request to CatalogD, get the latest compaction event Id from HMS, compare it with what is cached in CatalogD, and based on that decide whether to serve the data from cache or to refresh it from the filesystem. This can avoid notification based cache invalidation.

Also, since there will be an open txn for the current long running query which is being served from CatalogD, we can be sure that current file-metadata being served is not already deleted by the cleaner.

This proposal will use a new HMS API (https://issues.apache.org/jira/browse/HIVE-24828) to get the latest compaction id for a table.