[FLINK-11476] Create CatalogManager to manage multiple catalogs and encapsulate Calcite schema

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.9.0
• Component/s: Table SQL / API
• Labels: pull-request-available
• Assignee: Dawid Wysakowicz
• Reporter: Xuefu Zhang
• Created: 31/Jan/19 00:17
• Updated: 07/Jun/19 21:24
• Resolved: 24/May/19 09:34

Description

Flink allows for more than one registered catalogs. CatalogManager class is the holding class to manage and encapsulate the catalogs and their interrelations with Calcite.

Following section describes how table resolution should work:

PATH resolution:

First look into DEFAULT PATH: cat or cat.db. Then in the root. This is also the behavior of Calcite. We should mimic this behavior in Flink.
Example:

root:
  |- builtin
      |- default
          |- tab1
          |- tab2
      |- db1
          |- tab1
      |- clashing
          |- tab1
  |- cat1
      |- db1
          |- tab1
          |- tab2
  |- extCat1
      |- tab1
      |- clashing
          |- tab1
      |- extCat2
          |- tab1
          |- tab2
  |- clashing (ExternalCatalog)
      |- tab1

There is always a default catalog, initially set to builtin and default database initially set to default.

Assume 
default path = builtin then
  default.tab1 = builtin.default.tab1
  tab1 = error
  cat1.db1.tab1 = cat1.db1.tab1
  clashing.tab1 = builtin.clashing.tab1
default path = extCat1 then
  tab1 = extCat1.tab1
  clashing.tab1 = extCat1.clashing.tab1
default path = extCat1.extCat2 (do not support further nesting) then
  tab1 = extCat1.extCat2.tab1
  clashing.tab1 = clashing.tab1

Structure in the Planner(Calcite-specific)

root: CatalogManagerSchema(CatalogManager)
     |- CatalogCalciteSchema(ReadableCatalog)
         |- DatabaseCalciteSchema (ReadableCatalog scoped to DB)
             |- Table
     |- ExternalCatalogSchema
         |- Table
         |- ExternalCatalogSchema
             |- Table

Structure in the API

    
CatalogManager:
    |- ReadableCatalog
      |- CatalogTable
    |- ExternalCatalog
      |- ExternalCatalog
      |- ExternalCatalogTable

Issue Links

• blocks: FLINK-9172 Support catalogs in SQL-Client yaml config file
• is duplicated by: FLINK-10698 Create CatalogManager class manages all external catalogs and temporary meta objects
• is duplicated by: FLINK-10768 Move external catalog related code from TableEnvironment to CatalogManager
• is duplicated by: FLINK-10796 Add a default external catalog (as FlinkInMemoryCatalog) to CatalogManager
• links to: GitHub Pull Request #8214
• links to: GitHub Pull Request #8404