Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-11275 Unified Catalog APIs
  3. FLINK-11476

Create CatalogManager to manage multiple catalogs and encapsulate Calcite schema

    XMLWordPrintableJSON

    Details

      Description

      Flink allows for more than one registered catalogs. CatalogManager class is the holding class to manage and encapsulate the catalogs and their interrelations with Calcite.

      Following section describes how table resolution should work:

      PATH resolution:

      First look into DEFAULT PATH: cat or cat.db. Then in the root. This is also the behavior of Calcite. We should mimic this behavior in Flink.
      Example:

      root:
        |- builtin
            |- default
                |- tab1
                |- tab2
            |- db1
                |- tab1
            |- clashing
                |- tab1
        |- cat1
            |- db1
                |- tab1
                |- tab2
        |- extCat1
            |- tab1
            |- clashing
                |- tab1
            |- extCat2
                |- tab1
                |- tab2
        |- clashing (ExternalCatalog)
            |- tab1
      

      There is always a default catalog, initially set to builtin and default database initially set to default.

      Assume 
      default path = builtin then
        default.tab1 = builtin.default.tab1
        tab1 = error
        cat1.db1.tab1 = cat1.db1.tab1
        clashing.tab1 = builtin.clashing.tab1
      default path = extCat1 then
        tab1 = extCat1.tab1
        clashing.tab1 = extCat1.clashing.tab1
      default path = extCat1.extCat2 (do not support further nesting) then
        tab1 = extCat1.extCat2.tab1
        clashing.tab1 = clashing.tab1
      

      Structure in the Planner(Calcite-specific)

      root: CatalogManagerSchema(CatalogManager)
           |- CatalogCalciteSchema(ReadableCatalog)
               |- DatabaseCalciteSchema (ReadableCatalog scoped to DB)
                   |- Table
           |- ExternalCatalogSchema
               |- Table
               |- ExternalCatalogSchema
                   |- Table
      

      Structure in the API

          
      CatalogManager:
          |- ReadableCatalog
            |- CatalogTable
          |- ExternalCatalog
            |- ExternalCatalog
            |- ExternalCatalogTable
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                dwysakowicz Dawid Wysakowicz
                Reporter:
                xuefuz Xuefu Zhang
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 40m
                  40m