[FLINK-12251] Rework the Table API & SQL type system

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: Table SQL / API
• Labels: auto-deprioritized-major auto-unassigned
• Assignee: Unassigned
• Reporter: Timo Walther
• Created: 18/Apr/19 10:16
• Updated: 15/Aug/22 22:38

Description

Currently, the Table & SQL API relies on Flink’s TypeInformation at different positions in the code base. The API uses it for conversion between DataSet/DataStream API, casting, and table schema representation. The planning for code generation and serialization of runtime operators.

The past has shown that TypeInformation is useful when converting between DataSet/DataStream API, however, it does not integrate nicely with SQLs type system and depends on the programming language that is used.

There is consensus to perform a big rework of the type system with a better long-term vision and closer semantics to other SQL vendors and the standard itself.

FLIP-37 discusses the mid-term and long-term plan for supported types and their semantics.

https://docs.google.com/document/d/1a9HUb6OaBIoj9IRfbILcMFPrOL7ALeZ3rVI66dvA2_U/edit#

Issue Links

• is related to: FLINK-12710 Unify built-in and user-defined functions in the API modules
• relates to: FLINK-12457 Remove char support in blink
• relates to: FLINK-13773 Rework of the Expression Design
• mentioned in: Page Loading...

Sub-Tasks

• Setup a class hierarchy for the new logical type system: Closed
• Add the user-facing classes of the new type system: Closed
• Remove row interval type: Closed
• Fix ANY type serialization: Closed
• Improve the semantics of zero length character strings: Closed
• Expose the new type system through the API: Closed
• Replace InternalType with LogicalType in blink runner: Closed
• Introduce basic type inference interfaces: Closed
• Add a casting utility: Closed
• Add a type parser utility: Closed
• Add predefined type validators, strategies, and transformations: Closed
• Add documentation for the new Table & SQL API type system: Closed
• Bring the SQL CREATE TABLE DDL closer to FLIP-37: Closed
• Support List as a conversion class for ARRAY: Closed
• Update MapView and ListView interface to new type system: Closed
• Relax structured types constraints: Closed
• Rename ANY type to RAW type: Closed
• Preserve nullability for nested types: Closed
• Fully support RAW types in the API: Closed
• Support RAW types through the stack: Closed
• Support the NULL type for function calls: Closed
• Support inline structured types: Closed
• Don't allow self referencing structured type: Closed
• Simplify the creation of explicit structured types: Closed
• Revisit FlinkTypeFactory: Open
• Align Calcite's and Flink's SYMBOL types: Closed
• CREATE TABLE does not support interval types: Open
• Improve FieldsDataType: Closed
• Precision of TIME type does not work correctly: In Progress
• Cannot convert String or boxed-primitive arrays to DataStream using TypeInformation: Open
• Avro format logical type conversions do not work due to type mismatch: Closed
• Cannot set LocalDateTime column as rowtime when converting DataStream to Table: Open
• [FLIP-154] Enable type coercion: Open
• DataTypeExtractor does not check for default constructor: Closed
• Support for comparing instance of all types: Open
• Update data type documentation for 1.11: Closed
• Remove RowDataTypeInfo: Closed
• Update Table API set operations to the new type system: Closed
• Calling ROW() in a UDF results in UnsupportedOperationException: Closed
• Retain the column name when converting a Table to a DataStream: Open
• Type mismatch when using ROW in computed column: Reopened
• Cannot create empty array using ARRAY[]: Open
• Update TableResult.collect()/TableResult.print() to the new type system: Closed
• Support TIMESTAMP WITH TIME ZONE: Open
• GenericTypeInfo equality issue: Open
• DataTypes implementation for flink-csv with ZonedDateTime support: Open
• Add a union type: Open
• Convert CharSequence to String when registering / converting a Table: Open
• Add native Avro type support to the Table API & SQL: Open
• The precision in document and code of `INTERVAL DAY(p1) TO SECOND(p2)` are inconsistent: Open
• Failed to insert nested types using value construction functions: Closed
• Make checking for type root and family less verbose: Closed
• Update ReplicateRows to the new type system: Closed
• Use HashMap for MAP value constructors: Closed
• Logical type ROW lost inner field's nullability after convert to RelDataType: Closed