Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-7127

Fetch-on-demand metadata for the impalad-side catalog

    XMLWordPrintableJSON

    Details

    • Epic Color:
      ghx-label-5

      Description

      This is an umbrella JIRA to correspond to the design doc that proposes implementing the Impala catalog by managing a granular cache within the impalad. Individual JIRAs will be filed as subtasks.

        Attachments

          Issue Links

          1.
          Extract interfaces for frontend interaction with catalog objects Sub-task Resolved Todd Lipcon
          2.
          Support external data sources without catalogd Sub-task Open Unassigned
          3.
          Skeleton implementation of LocalCatalog Sub-task Resolved Todd Lipcon
          4.
          Support configuring impalad to use the LocalCatalog Sub-task Resolved Todd Lipcon
          5.
          Build out support for HDFS tables and views in LocalCatalog Sub-task Resolved Todd Lipcon
          6.
          Extract interfaces for partition pruning prior to fetching partitions Sub-task Resolved Todd Lipcon
          7.
          Support functions in LocalCatalog Sub-task Resolved Todd Lipcon
          8.
          Support DDL in LocalCatalog using existing catalogd Sub-task Resolved Todd Lipcon
          9.
          Support querying Kudu tables in LocalCatalog Sub-task Resolved Todd Lipcon
          10.
          Support querying HBase tables in LocalCatalog Sub-task Resolved Tianyi Wang
          11.
          Support CREATE TABLE AS SELECT with LocalCatalog Sub-task Resolved Todd Lipcon
          12.
          Support for INSERT and LOAD DATA in LocalCatalog Sub-task Resolved Todd Lipcon
          13.
          Support TABLESAMPLE and stats extrapolation in LocalCatalog Sub-task Resolved Todd Lipcon
          14.
          Support Avro tables in LocalCatalog Sub-task Resolved Todd Lipcon
          15.
          Fetch table and partition metadata from catalogd Sub-task Resolved Todd Lipcon
          16.
          Simple granular caching of partition metadata in impalad Sub-task Resolved Todd Lipcon
          17.
          Automated test for concurrent DDL and metadata queries Sub-task Resolved Unassigned
          18.
          Size-based eviction for LocalCatalog LRU cache Sub-task Resolved Todd Lipcon
          19.
          Invalidate local catalog cache based on topic updates Sub-task Resolved Todd Lipcon
          20.
          impalad should wait for catalogd during start up Sub-task Resolved Bharath Vissapragada
          21.
          Slim down metastore Partition objects in LocalCatalog cache Sub-task Resolved Quanlong Huang
          22.
          Support global INVALIDATE METADATA on fetch-on-demand impalad Sub-task Resolved Quanlong Huang
          23.
          Clean up user-facing error messages in LocalCatalog mode Sub-task Open Unassigned
          24.
          Create table after drop can lead to table not found exception Sub-task Resolved Vuk Ercegovac
          25.
          Support sentry roles/privileges with LocalCatalog Sub-task Resolved Todd Lipcon
          26.
          Expose fetch-from-catalogd cache and latency metrics in profiles Sub-task Resolved Vuk Ercegovac
          27.
          Re-plan queries on InconsistentMetadataException Sub-task Resolved Vuk Ercegovac
          28.
          Add daemon-level metrics about fetch-from-catalog cache Sub-task Resolved Bharath Vissapragada
          29.
          Add retry/back-off to fetch-from-catalog RPCs Sub-task Resolved Tianyi Wang
          30.
          Optimize fetch-from-catalog by caching partitions across table versions Sub-task Resolved Quanlong Huang
          31.
          Handle invalidation races in CatalogdMetaProvider cache Sub-task Resolved Todd Lipcon
          32.
          CatalogdMetaProvider should fetch incremental stats data on-demand Sub-task Open Unassigned
          33.
          Re-evaluate Avro/Kudu/HBase table performance with fetch-from-catalogd Sub-task Open Unassigned
          34.
          Support HDFS caching with LocalCatalog Sub-task Resolved Quanlong Huang
          35.
          Support HDFS permissions checks with LocalCatalog Sub-task Open Unassigned
          36.
          Consider using read-write locks for partial fetch requests. Sub-task Open Unassigned
          37.
          Don't need to acquire table locks in gathering catalog topic updates in minimal topic mode Sub-task Closed Unassigned
          38.
          Only send invalidations in DDL responses to LocalCatalog coordinators Sub-task Resolved Quanlong Huang

            Activity

              People

              • Assignee:
                tlipcon Todd Lipcon
                Reporter:
                tlipcon Todd Lipcon
              • Votes:
                0 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                • Created:
                  Updated: