Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-11275

Unified Catalog APIs

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Table SQL / API
    • Labels:
      None

      Description

      During Flink-Hive integration, we found that the current Catalog APIs are quite cumbersome and to a large extent requiring significant rework. While previous APIs are essentially not used, at least not in Flink code base, we needs to be careful in defining a new set of APIs to avoid future rework again.

      This is an uber JIRA covering all the work outlined in FLIP-30.

        Attachments

          Issue Links

          1.
          Add ReadableCatalog, ReadableWritableCatalog, and other related interfaces Sub-task Closed Xuefu Zhang
          2.
          Adapt existing InMemoryExternalCatalog to GenericInMemoryCatalog Sub-task Closed Bowen Li  
          3.
          Create CatalogManager to manage multiple catalogs and encapsulate Calcite schema Sub-task Resolved Dawid Wysakowicz
          4.
          Define catalog entries in SQL client YAML file and handle the creation and registration of those entries Sub-task Closed Bowen Li  
          5.
          Handle existing table registration via YAML file in the context of catalog support Sub-task Open Xuefu Zhang  
          6.
          Create HiveTableFactory that creates TableSource/Sink from a Hive table Sub-task Closed Xuefu Zhang
          7.
          Integrate HiveTableFactory with existing table factory discovery mechanism Sub-task Open Xuefu Zhang  
          8.
          Add partition related catalog APIs Sub-task Closed Bowen Li
          9.
          Add function related catalog APIs Sub-task Closed Xuefu Zhang
          10.
          Rename a few exception class names that were migrated from scala Sub-task Open Xuefu Zhang  
          11.
          Convert CatalogTable to org.apache.calcite.schema.Table so that planner can use unified catalog APIs Sub-task Closed Dawid Wysakowicz
          12.
          Decouple CatalogManager from Calcite Sub-task Closed Dawid Wysakowicz  
          13.
          Add documentation for catalog Sub-task Closed Bowen Li
          14.
          Add documentation for catalog entries in SQL client YAML file Sub-task Closed Bowen Li  
          15.
          Add stats related catalog APIs Sub-task Closed Xuefu Zhang
          16.
          Clean up Catalog APIs to make them more consistent and coherent Sub-task Closed Xuefu Zhang
          17.
          Add more detailed javadoc for getDescription() and getDetailedDescription() in catalog object interfaces Sub-task In Progress Bowen Li
          18.
          Support partitioned view in catalog API Sub-task Open Unassigned  
          19.
          Unify ReadableCatalog and ReadableWritableCatalog interfaces to Catalog interface Sub-task Closed Bowen Li
          20.
          Add column stats for decimal type Sub-task Open Unassigned  
          21.
          Add some more (negative) test cases for FLINK-12365 Sub-task Open Unassigned  
          22.
          alterTable() should ensure existing base table and the new one are of the same type Sub-task Closed Bowen Li
          23.
          Clean up catalog API on default/current DB Sub-task Closed Xuefu Zhang
          24.
          Combine HiveCatalog and GenericHiveMetastoreCatalog Sub-task Closed Xuefu Zhang
          25.
          Alteration APIs in catalogs should check existing object and new object are of the same class Sub-task Closed Bowen Li
          26.
          Add AbstractCatalog to manage the common catalog name and default database name for catalogs Sub-task Closed Bowen Li  
          27.
          deprecate ExternalCatalog and its subclasses and impls Sub-task Closed Bowen Li
          28.
          Unify catalog meta-objects implementations Sub-task Closed Bowen Li  
          29.
          Unify catalog database implementations Sub-task Closed Bowen Li
          30.
          serialize catalog table to properties for table discovery service Sub-task Closed Bowen Li
          31.
          Convert CatalogView to org.apache.calcite.schema.Table so that planner can use unified catalog APIs Sub-task Open Dawid Wysakowicz
          32.
          unify GenericCatalogTable, HiveCatalogTable and AbstractCatalogTable into CatalogTableImpl Sub-task Closed Bowen Li
          33.
          unify catalog view implementations Sub-task Closed Bowen Li
          34.
          restore AbstractCatalogTable as parent for CatalogTableImpl and ConnectorCatalogTable Sub-task Closed Bowen Li
          35.
          Generate HiveTableSink from from a Hive table Sub-task Closed Xuefu Zhang
          36.
          unify catalog partition implementations Sub-task Closed Bowen Li
          37.
          unify catalog function implementations Sub-task Closed Bowen Li
          38.
          Generate HiveTableSource from from a Hive table Sub-task Closed Xuefu Zhang
          39.
          Fix the Optional.orElse() usage issue in DatabaseCalciteSchema Sub-task Closed Xuefu Zhang
          40.
          add Chinese documentation for catalogs Sub-task Open frank wang
          41.
          create CatalogTableBuilder to support building CatalogTable from descriptors Sub-task Closed Bowen Li

            Activity

              People

              • Assignee:
                xuefuz Xuefu Zhang
                Reporter:
                xuefuz Xuefu Zhang
              • Votes:
                0 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 11h 10m
                  11h 10m