[FLINK-10618] Introduce catalog for Flink tables

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Done
• Affects Version/s: 1.6.1
• Component/s: Table SQL / API, Table SQL / Client
• Assignee: Xuefu Zhang
• Reporter: Xuefu Zhang
• Created: 19/Oct/18 23:37
• Updated: 02/May/19 04:42
• Resolved: 02/May/19 04:42

Description

This JIRA covers the 2nd aspect of Flink-Hive metastore integration.

Besides meta objects such as tables that may come from an ExternalCatalog, Flink also deals with tables/views/functions that are created on the fly (in memory), or specified in a configuration file. Those objects don't belong to any ExternalCatalog, yet Flink either stores them in memory, which are non-persistent, or recreates them from a file, which is a big pain for the user. Those objects are only known to Flink but Flink has a poor management for them.

Since they are typical objects in a database catalog, it's natural to have a catalog that manages those objects. The interface will be similar to ExternalCatalog, which contains meta objects that are not managed by Flink. There are several possible implementations of the Flink internal catalog interface: memory, file, external registry (such as confluent schema registry or Hive metastore), and relational database, etc.

The initial functionality as well as the catalog hierarchy could be very simple. The basic functionality of the catalog will be mostly create, alter, and drop tables, views, functions, etc. Obviously, this can evolve over the time.

We plan to provide implementations: in-memory and in Hive metastore.

Issue Links

• is a parent of: FLINK-10697 Create FlinkInMemoryCatalog class, an in-memory catalog that stores Flink's meta objects for production use
• is a parent of: FLINK-10699 Create FlinkHmsCatalog for persistent Flink meta objects using Hive metastore as a registry
• links to: Google doc for the design