[FLINK-10744] Integrate Flink with Hive metastore

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Implemented
• Affects Version/s: 1.6.2
• Component/s: Connectors / Hive, Table SQL / Ecosystem
• Assignee: Xuefu Zhang
• Reporter: Xuefu Zhang
• Created: 31/Oct/18 22:10
• Updated: 29/Apr/21 08:44
• Resolved: 29/Apr/21 08:44

Description

This JIRA keeps track of the effort of FLINK-10556 on Hive metastore integration. It mainly covers two aspects:

1. Register Hive metastore as an external catalog of Flink, such that Hive table metadata can be accessed directly.
2. Store Flink metadata (tables, views, UDFs, etc) in a catalog that utilizes Hive as the schema registry.

Discussions and resulting design doc will be shared here, but detailed work items will be tracked by sub-tasks.

Issue Links

• is a child of: FLINK-10556 Integration with Apache Hive
• relates to: FLINK-10686 Introduce a flink-table-common module
• links to: Google doc for the design

Sub-Tasks

• Implement an external catalog for Hive: Closed
• Introduce catalog for Flink tables: Resolved
• Add APIs to ExternalCatalog for views and UDFs: Closed
• Create FlinkInMemoryCatalog class, an in-memory catalog that stores Flink's meta objects for production use: Closed
• Create FlinkHmsCatalog for persistent Flink meta objects using Hive metastore as a registry: Closed
• Create CatalogManager class manages all external catalogs and temporary meta objects: Closed
• Refactor TableEnvironment so that all registration calls delegate to CatalogManager: Resolved
• Adapt SQL-client configuration file to specify external catalogs and default catalog: Closed
• Create a command line tool to migrate meta objects specified in SQL client configuration: Closed
• Move external catalog related code from TableEnvironment to CatalogManager: Closed
• port InMemoryExternalCatalog to java: Closed
• Add a default external catalog (as FlinkInMemoryCatalog) to CatalogManager: Closed
• Enhance SQL-Client to recursively list UDFs: Closed
• Enhance TableEnvironment to connect to a catalog via a descriptor: Closed
• Enhance SQL-Client to recursively list tables: Closed
• Update documentation for catalogs in SQL-Client: Closed
• Fix DESCRIBE command to support catalog tables: Closed