[FLINK-11275] Unified Catalog APIs

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Table SQL / API
• Assignee: Xuefu Zhang
• Reporter: Xuefu Zhang
• Created: 07/Jan/19 23:52
• Updated: 05/Jun/19 21:52

Description

During Flink-Hive integration, we found that the current Catalog APIs are quite cumbersome and to a large extent requiring significant rework. While previous APIs are essentially not used, at least not in Flink code base, we needs to be careful in defining a new set of APIs to avoid future rework again.

This is an uber JIRA covering all the work outlined in FLIP-30.

Issue Links

• contains: FLINK-12753 Implement GenericInMemoryCatalog
• contains: FLINK-12755 implement HiveCatalog
• contains: FLINK-12256 Implement KafkaReadableCatalog
• contains: FLINK-12278 Support user implemented and customized catalogs thru dynamic class loading
• is related to: FLINK-12625 Support catalogs in SQL Client

Sub-Tasks

• Add ReadableCatalog, ReadableWritableCatalog, and other related interfaces: Closed
• Adapt existing InMemoryExternalCatalog to GenericInMemoryCatalog: Closed
• Create CatalogManager to manage multiple catalogs and encapsulate Calcite schema: Resolved
• Define catalog entries in SQL client YAML file and handle the creation and registration of those entries: Closed
• Handle existing table registration via YAML file in the context of catalog support: Open
• Create HiveTableFactory that creates TableSource/Sink from a Hive table: Closed
• Integrate HiveTableFactory with existing table factory discovery mechanism: Open
• Add partition related catalog APIs: Closed
• Add function related catalog APIs: Closed
• Rename a few exception class names that were migrated from scala: Open
• Convert CatalogTable to org.apache.calcite.schema.Table so that planner can use unified catalog APIs: Closed
• Decouple CatalogManager from Calcite: Closed
• Add documentation for catalog: Closed
• Add documentation for catalog entries in SQL client YAML file: Closed
• Add stats related catalog APIs: Closed
• Clean up Catalog APIs to make them more consistent and coherent: Closed
• Add more detailed javadoc for getDescription() and getDetailedDescription() in catalog object interfaces: In Progress
• Support partitioned view in catalog API: Open
• Unify ReadableCatalog and ReadableWritableCatalog interfaces to Catalog interface: Closed
• Add column stats for decimal type: Open
• Add some more (negative) test cases for FLINK-12365: Open
• alterTable() should ensure existing base table and the new one are of the same type: Closed
• Clean up catalog API on default/current DB: Closed
• Combine HiveCatalog and GenericHiveMetastoreCatalog: Closed
• Alteration APIs in catalogs should check existing object and new object are of the same class: Closed
• Add AbstractCatalog to manage the common catalog name and default database name for catalogs: Closed
• deprecate ExternalCatalog and its subclasses and impls: Closed
• Unify catalog meta-objects implementations: Closed
• Unify catalog database implementations: Closed
• serialize catalog table to properties for table discovery service: Closed
• Convert CatalogView to org.apache.calcite.schema.Table so that planner can use unified catalog APIs: Open
• unify GenericCatalogTable, HiveCatalogTable and AbstractCatalogTable into CatalogTableImpl: Closed
• unify catalog view implementations: Closed
• restore AbstractCatalogTable as parent for CatalogTableImpl and ConnectorCatalogTable: Closed
• Generate HiveTableSink from from a Hive table: Closed
• unify catalog partition implementations: Closed
• unify catalog function implementations: Closed
• Generate HiveTableSource from from a Hive table: Closed
• Fix the Optional.orElse() usage issue in DatabaseCalciteSchema: Closed
• add Chinese documentation for catalogs: Open
• create CatalogTableBuilder to support building CatalogTable from descriptors: Closed