[IMPALA-7127] Fetch-on-demand metadata for the impalad-side catalog

• Type: New Feature
• Status: Reopened
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: Impala 3.2.0
• Component/s: Catalog, Frontend
• Labels: catalog-v2
• Epic Color: ghx-label-5
• Assignee: Todd Lipcon
• Reporter: Todd Lipcon
• Created: 06/Jun/18 01:18
• Updated: 25/May/22 03:38

Description

This is an umbrella JIRA to correspond to the design doc that proposes implementing the Impala catalog by managing a granular cache within the impalad. Individual JIRAs will be filed as subtasks.

Issue Links

• is duplicated by: IMPALA-3547 Bound the size of Impalad catalog cache
• is related to: IMPALA-8822 No metadata loading information in query profile when Catalog V2 enabled
• is related to: IMPALA-9823 use_local_catalog and related flags shouldn't be hidden
• is related to: IMPALA-7615 Partition metadata mismatch should be handled gracefully in local catalog mode.
• is related to: IMPALA-4045 Catalog cache update should not tied to statestore update frequency
• relates to: IMPALA-9669 loaded views are still returned as tables for GET_TABLES in LocalCatalog mode
• relates to: IMPALA-8675 Remove db/table count metrics from impalad in LocalCatalog mode
• relates to: IMPALA-9817 Document flags of Fetch-on-demand metadata coordinators

Sub-Tasks

• Clean up user-facing error messages in LocalCatalog mode: Open
• CatalogdMetaProvider should fetch incremental stats data on-demand: Open
• Re-evaluate Avro/Kudu/HBase table performance with fetch-from-catalogd: Open
• Support HDFS permissions checks with LocalCatalog: Open