[IMPALA-7127] Fetch-on-demand metadata for the impalad-side catalog

• Type: New Feature
• Status: Reopened
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: Impala 3.2.0
• Component/s: Catalog, Frontend
• Labels: catalog-v2
• Epic Color: ghx-label-5
• Assignee: Todd Lipcon
• Reporter: Todd Lipcon
• Created: 06/Jun/18 01:18
• Updated: 25/May/22 03:38

Description

This is an umbrella JIRA to correspond to the design doc that proposes implementing the Impala catalog by managing a granular cache within the impalad. Individual JIRAs will be filed as subtasks.

Issue Links

• is duplicated by: IMPALA-3547 Bound the size of Impalad catalog cache
• is related to: IMPALA-8822 No metadata loading information in query profile when Catalog V2 enabled
• is related to: IMPALA-9823 use_local_catalog and related flags shouldn't be hidden
• is related to: IMPALA-7615 Partition metadata mismatch should be handled gracefully in local catalog mode.
• is related to: IMPALA-4045 Catalog cache update should not tied to statestore update frequency
• relates to: IMPALA-9669 loaded views are still returned as tables for GET_TABLES in LocalCatalog mode
• relates to: IMPALA-8675 Remove db/table count metrics from impalad in LocalCatalog mode
• relates to: IMPALA-9817 Document flags of Fetch-on-demand metadata coordinators

Sub-Tasks

• Extract interfaces for frontend interaction with catalog objects: Resolved
• Support external data sources in local catalog mode: Resolved
• Skeleton implementation of LocalCatalog: Resolved
• Support configuring impalad to use the LocalCatalog: Resolved
• Build out support for HDFS tables and views in LocalCatalog: Resolved
• Extract interfaces for partition pruning prior to fetching partitions: Resolved
• Support functions in LocalCatalog: Resolved
• Support DDL in LocalCatalog using existing catalogd: Resolved
• Support querying Kudu tables in LocalCatalog: Resolved
• Support querying HBase tables in LocalCatalog: Resolved
• Support CREATE TABLE AS SELECT with LocalCatalog: Resolved
• Support for INSERT and LOAD DATA in LocalCatalog: Resolved
• Support TABLESAMPLE and stats extrapolation in LocalCatalog: Resolved
• Support Avro tables in LocalCatalog: Resolved
• Fetch table and partition metadata from catalogd: Resolved
• Simple granular caching of partition metadata in impalad: Resolved
• Automated test for concurrent DDL and metadata queries: Resolved
• Size-based eviction for LocalCatalog LRU cache: Resolved
• Invalidate local catalog cache based on topic updates: Resolved
• impalad should wait for catalogd during start up: Resolved
• Slim down metastore Partition objects in LocalCatalog cache: Resolved
• Support global INVALIDATE METADATA on fetch-on-demand impalad: Resolved
• Clean up user-facing error messages in LocalCatalog mode: Open
• Create table after drop can lead to table not found exception: Resolved
• Support sentry roles/privileges with LocalCatalog: Resolved
• Expose fetch-from-catalogd cache and latency metrics in profiles: Resolved
• Re-plan queries on InconsistentMetadataException: Resolved
• Add daemon-level metrics about fetch-from-catalog cache: Resolved
• Add retry/back-off to fetch-from-catalog RPCs: Resolved
• Optimize fetch-from-catalog by caching partitions across table versions: Resolved
• Handle invalidation races in CatalogdMetaProvider cache: Resolved
• CatalogdMetaProvider should fetch incremental stats data on-demand: Open
• Re-evaluate Avro/Kudu/HBase table performance with fetch-from-catalogd: Open
• Support HDFS caching with LocalCatalog: Resolved
• Support HDFS permissions checks with LocalCatalog: Open
• Consider using read-write locks for partial fetch requests.: Resolved
• Don't need to acquire table locks in gathering catalog topic updates in minimal topic mode: Closed
• Only send invalidations in DDL responses to LocalCatalog coordinators: Resolved