Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-10744

Integrate Flink with Hive metastore

    XMLWordPrintableJSON

    Details

      Description

      This JIRA keeps track of the effort of FLINK-10556 on Hive metastore integration. It mainly covers two aspects:

      1. Register Hive metastore as an external catalog of Flink, such that Hive table metadata can be accessed directly.
      2. Store Flink metadata (tables, views, UDFs, etc) in a catalog that utilizes Hive as the schema registry.

      Discussions and resulting design doc will be shared here, but detailed work items will be tracked by sub-tasks.

        Attachments

          Issue Links

          1.
          Implement an external catalog for Hive Sub-task Closed Xuefu Zhang  
          2.
          Introduce catalog for Flink tables Sub-task Resolved Xuefu Zhang  
          3.
          Add APIs to ExternalCatalog for views and UDFs Sub-task Closed Bowen Li
          4.
          Create FlinkInMemoryCatalog class, an in-memory catalog that stores Flink's meta objects for production use Sub-task Closed Bowen Li  
          5.
          Create FlinkHmsCatalog for persistent Flink meta objects using Hive metastore as a registry Sub-task Closed Bowen Li  
          6.
          Create CatalogManager class manages all external catalogs and temporary meta objects Sub-task Closed Bowen Li  
          7.
          Refactor TableEnvironment so that all registration calls delegate to CatalogManager Sub-task Resolved Xuefu Zhang  
          8.
          Adapt SQL-client configuration file to specify external catalogs and default catalog Sub-task Closed Xuefu Zhang  
          9.
          Create a command line tool to migrate meta objects specified in SQL client configuration Sub-task Open Xuefu Zhang  
          10.
          Move external catalog related code from TableEnvironment to CatalogManager Sub-task Closed Bowen Li
          11.
          port InMemoryExternalCatalog to java Sub-task Closed Bowen Li
          12.
          Add a default external catalog (as FlinkInMemoryCatalog) to CatalogManager Sub-task Closed Bowen Li  
          13.
          Enhance SQL-Client to recursively list UDFs Sub-task Open vinoyang  
          14.
          Enhance TableEnvironment to connect to a catalog via a descriptor Sub-task Open Unassigned
          15.
          Enhance SQL-Client to recursively list tables Sub-task Open Unassigned  
          16.
          Update documentation for catalogs in SQL-Client Sub-task Open Unassigned  
          17.
          Fix DESCRIBE command to support catalog tables Sub-task Open vinoyang

            Activity

              People

              • Assignee:
                xuefuz Xuefu Zhang
                Reporter:
                xuefuz Xuefu Zhang
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h
                  1h