[FLINK-21394] FLIP-164: Improve Schema Handling in Catalogs

• Type: Improvement
• Status: Open
• Priority: Not a Priority
• Resolution: Unresolved
• Component/s: Table SQL / API
• Labels: auto-deprioritized-major auto-deprioritized-minor auto-unassigned
• Assignee: Unassigned
• Reporter: Timo Walther
• Created: 18/Feb/21 11:10
• Updated: 04/Apr/22 22:38

Description

Schema information is necessary at different locations in the Table API for defining tables and/or views. In particular, it is necessary to define the schema in a programmatic DDL (FLIP-129) and when converting a DataStream to a Table (FLIP-136).

We need similar APIs in the Catalog interfaces such that catalog implementations can define table/views in a unified way.

This FLIP updates the class hierarchy to achieve the following goals:

• make it visible whether a schema is resolved or unresolved and when the resolution happens
• offer a unified API for FLIP-129, FLIP-136, and catalogs
• allow arbitrary data types and expressions in the schema for watermark spec or columns
• have access to other catalogs for declaring a data type or expression via CatalogManager
• cleaned up TableSchema
• remain backwards compatible in the persisted properties and API

Issue Links

• mentioned in: Page Loading...

Sub-Tasks

• Add a SqlExpression in table-common: Closed
• Implement Schema, ResolvedSchema, SchemaResolver: Closed
• Implement ResolvedExpression.asSerializableString for SQL: Closed
• Update the Catalog APIs: Closed
• Update use new schema in Table, TableResult, TableOperation: Closed
• Update DynamicTableFactory.Context to use ResolvedCatalogTable: Closed
• Reduce usage of TableSchema in the planner: Closed
• Introduce Schema and ResolvedSchema in Python API: Open
• Reduce usage of CatalogViewImpl in planner: Resolved
• Drop the deprecated CatalogViewImpl: Open
• Reduce usage of CatalogTableImpl in planner: Closed