[IMPALA-3681] Automatic invalidation of HMS metadata using Hive replication API (HIVE-7973)

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: Impala 2.6.0
• Component/s: Catalog
• Labels: catalog-server
• Target Version: Product Backlog
• Assignee: Unassigned
• Reporter: Dimitris Tsirogiannis
• Created: 06/Jun/16 17:33
• Updated: 30/Jan/19 00:02
• Resolved: 30/Jan/19 00:02

Description

Hive is adding support for a replication API (https://issues.apache.org/jira/browse/HIVE-7973) that allows an HMS client to retrieve a global stream of HMS modifications made by all the HMS processes (in case of HMS HA). The Catalog server could use that API to automatically collect and update cached HMS metadata, thus eliminating the need for the user to call invalidate and refresh. This can only handle HMS metadata; file and block metadata will still need to be updated from the Namenode (e.g. by using inotify).

Issue Links

• duplicates: IMPALA-7970 Add support for automatic invalidates by polling metastore events
• is related to: IMPALA-555 Automatically perform REFRESH/INVALIDATE METADATA for users
• relates to: IMPALA-4272 Automatic invalidation HDFS file and block metadata