[IMPALA-3124] Add external metadata notification mechanisms to Impala

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: Impala 2.2.4
• Component/s: Catalog
• Labels: catalog-server
• Target Version: Product Backlog
• Assignee: Unassigned
• Reporter: Dimitris Tsirogiannis
• Created: 03/Mar/16 00:37
• Updated: 23/Dec/20 19:29
• Resolved: 23/Dec/20 19:29

Description

Currently, users needs to run refresh/invalidate metadata to update metadata when data (files, blocks) and/or metadata (tables, partitions) are modified externally. These operation cause many consistency and performance issues. We should add external metadata notification mechanism (e.g. inotify or HMS listeners) so that the catalog server can automatically sync metadata as they change.

Issue Links

• duplicates: IMPALA-7954 Support automatic invalidates using metastore notification events

Sub-Tasks

• Automatic invalidation of HMS metadata using Hive replication API (HIVE-7973): Resolved