Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-12251

Rework the Table API & SQL type system

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Table SQL / API
    • Labels:
      None

      Description

      Currently, the Table & SQL API relies on Flinkā€™s TypeInformation at different positions in the code base. The API uses it for conversion between DataSet/DataStream API, casting, and table schema representation. The planning for code generation and serialization of runtime operators.

      The past has shown that TypeInformation is useful when converting between DataSet/DataStream API, however, it does not integrate nicely with SQLs type system and depends on the programming language that is used.

      There is consensus to perform a big rework of the type system with a better long-term vision and closer semantics to other SQL vendors and the standard itself.

      FLIP-37 discusses the mid-term and long-term plan for supported types and their semantics.

      https://docs.google.com/document/d/1a9HUb6OaBIoj9IRfbILcMFPrOL7ALeZ3rVI66dvA2_U/edit#

        Attachments

        Issue Links

        1.
        Setup a class hierarchy for the new logical type system Sub-task Closed Timo Walther Actions
        2.
        Add the user-facing classes of the new type system Sub-task Closed Timo Walther Actions
        3.
        Remove row interval type Sub-task Closed Timo Walther Actions
        4.
        Fix ANY type serialization Sub-task Closed Timo Walther Actions
        5.
        Improve the semantics of zero length character strings Sub-task Closed Timo Walther Actions
        6.
        Expose the new type system through the API Sub-task Closed Timo Walther Actions
        7.
        Replace InternalType with LogicalType in blink runner Sub-task Closed Jingsong Lee Actions
        8.
        Introduce basic type inference interfaces Sub-task Closed Timo Walther Actions
        9.
        Add a casting utility Sub-task Closed Timo Walther Actions
        10.
        Add a type parser utility Sub-task Closed Timo Walther Actions
        11.
        Add predefined type validators, strategies, and transformations Sub-task Closed Timo Walther Actions
        12.
        Add documentation for the new Table & SQL API type system Sub-task Closed Timo Walther Actions
        13.
        Bring the SQL CREATE TABLE DDL closer to FLIP-37 Sub-task Closed Danny Chen Actions
        14.
        Support List as a conversion class for ARRAY Sub-task Closed Timo Walther   Actions
        15.
        Update MapView and ListView interface to new type system Sub-task Closed Unassigned Actions
        16.
        Relax structured types constraints Sub-task Closed Timo Walther Actions
        17.
        Rename ANY type to RAW type Sub-task Closed Timo Walther Actions
        18.
        Preserve nullability for nested types Sub-task Closed Timo Walther Actions
        19.
        Fully support RAW types in the API Sub-task Closed Timo Walther Actions
        20.
        Support RAW types through the stack Sub-task Closed Timo Walther Actions
        21.
        Support the NULL type for function calls Sub-task Closed Timo Walther   Actions
        22.
        Support inline structured types Sub-task Closed Timo Walther   Actions
        23.
        Don't allow self referencing structured type Sub-task Closed Timo Walther   Actions
        24.
        Simplify the creation of explicit structured types Sub-task Closed Timo Walther   Actions
        25.
        Revisit FlinkTypeFactory Sub-task Open Unassigned   Actions
        26.
        Align Calcite's and Flink's SYMBOL types Sub-task Open Unassigned   Actions
        27.
        CREATE TABLE does not support interval types Sub-task Open Unassigned   Actions
        28.
        Improve FieldsDataType Sub-task Closed Timo Walther   Actions
        29.
        Precision of TIME type does not work correctly Sub-task In Progress Unassigned   Actions
        30.
        Support for casting collection types. Sub-task Open Unassigned   Actions
        31.
        Cannot convert String or boxed-primitive arrays to DataStream using TypeInformation Sub-task Open Unassigned   Actions
        32.
        Avro format logical type conversions do not work due to type mismatch Sub-task Closed Unassigned   Actions
        33.
        Cannot set LocalDateTime column as rowtime when converting DataStream to Table Sub-task Open Unassigned   Actions
        34.
        Enable type coercion Sub-task Open Unassigned   Actions
        35.
        DataTypeExtractor does not check for default constructor Sub-task Closed Timo Walther   Actions
        36.
        Support for comparing instance of all types Sub-task Open Unassigned   Actions
        37.
        Update data type documentation for 1.11 Sub-task Closed Timo Walther   Actions
        38.
        Remove RowDataTypeInfo Sub-task Closed Timo Walther   Actions
        39.
        Update Table API set operations to the new type system Sub-task Closed Timo Walther   Actions
        40.
        Calling ROW() in a UDF results in UnsupportedOperationException Sub-task Closed Timo Walther   Actions
        41.
        Retain the column name when converting a Table to a DataStream Sub-task Open Unassigned   Actions
        42.
        Type mismatch when using ROW in computed column Sub-task Open Unassigned   Actions
        43.
        Cannot create empty array using ARRAY[] Sub-task Open Unassigned   Actions
        44.
        Update TableResult.collect()/TableResult.print() to the new type system Sub-task Closed Timo Walther   Actions
        45.
        Support TIMESTAMP WITH TIME ZONE Sub-task Open Unassigned   Actions
        46.
        GenericTypeInfo equality issue Sub-task Open Unassigned   Actions
        47.
        DataTypes implementation for flink-csv with ZonedDateTime support Sub-task Open Unassigned   Actions
        48.
        Add a union type Sub-task Open Unassigned   Actions
        49.
        Convert CharSequence to String when registering / converting a Table Sub-task Open Unassigned   Actions
        50.
        Add native Avro type support to the Table API & SQL Sub-task Open Unassigned   Actions

          Activity

            People

            • Assignee:
              twalthr Timo Walther
              Reporter:
              twalthr Timo Walther

              Dates

              • Created:
                Updated:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 9h
                9h

                  Issue deployment