Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-12251

Rework the Table API & SQL type system

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Table SQL / API
    • Labels:

      Description

      Currently, the Table & SQL API relies on Flinkā€™s TypeInformation at different positions in the code base. The API uses it for conversion between DataSet/DataStream API, casting, and table schema representation. The planning for code generation and serialization of runtime operators.

      The past has shown that TypeInformation is useful when converting between DataSet/DataStream API, however, it does not integrate nicely with SQLs type system and depends on the programming language that is used.

      There is consensus to perform a big rework of the type system with a better long-term vision and closer semantics to other SQL vendors and the standard itself.

      FLIP-37 discusses the mid-term and long-term plan for supported types and their semantics.

      https://docs.google.com/document/d/1a9HUb6OaBIoj9IRfbILcMFPrOL7ALeZ3rVI66dvA2_U/edit#

        Attachments

          Issue Links

          1.
          Setup a class hierarchy for the new logical type system Sub-task Closed Timo Walther
          2.
          Add the user-facing classes of the new type system Sub-task Closed Timo Walther
          3.
          Remove row interval type Sub-task Closed Timo Walther
          4.
          Fix ANY type serialization Sub-task Closed Timo Walther
          5.
          Improve the semantics of zero length character strings Sub-task Closed Timo Walther
          6.
          Expose the new type system through the API Sub-task Closed Timo Walther
          7.
          Replace InternalType with LogicalType in blink runner Sub-task Closed Jingsong Lee
          8.
          Introduce basic type inference interfaces Sub-task Closed Timo Walther
          9.
          Add a casting utility Sub-task Closed Timo Walther
          10.
          Add a type parser utility Sub-task Closed Timo Walther
          11.
          Add predefined type validators, strategies, and transformations Sub-task Closed Timo Walther
          12.
          Add documentation for the new Table & SQL API type system Sub-task Closed Timo Walther
          13.
          Bring the SQL CREATE TABLE DDL closer to FLIP-37 Sub-task Closed Danny Chen
          14.
          Support List as a conversion class for ARRAY Sub-task Closed Timo Walther  
          15.
          Update MapView and ListView interface to new type system Sub-task Closed Unassigned
          16.
          Relax structured types constraints Sub-task Closed Timo Walther
          17.
          Rename ANY type to RAW type Sub-task Closed Timo Walther
          18.
          Preserve nullability for nested types Sub-task Closed Timo Walther
          19.
          Fully support RAW types in the API Sub-task Closed Timo Walther
          20.
          Support RAW types through the stack Sub-task Closed Timo Walther
          21.
          Support the NULL type for function calls Sub-task Closed Timo Walther  
          22.
          Support inline structured types Sub-task Closed Timo Walther  
          23.
          Don't allow self referencing structured type Sub-task Closed Timo Walther  
          24.
          Simplify the creation of explicit structured types Sub-task Closed Timo Walther  
          25.
          Revisit FlinkTypeFactory Sub-task Open Unassigned  
          26.
          Align Calcite's and Flink's SYMBOL types Sub-task Open Unassigned  
          27.
          CREATE TABLE does not support interval types Sub-task Open Unassigned  
          28.
          Improve FieldsDataType Sub-task Closed Timo Walther  
          29.
          Precision of TIME type does not work correctly Sub-task In Progress Unassigned  
          30.
          Support for casting collection types. Sub-task Open Unassigned  
          31.
          Cannot convert String or boxed-primitive arrays to DataStream using TypeInformation Sub-task Open Unassigned  
          32.
          Avro format logical type conversions do not work due to type mismatch Sub-task Closed Unassigned  
          33.
          Cannot set LocalDateTime column as rowtime when converting DataStream to Table Sub-task Open Unassigned  
          34.
          Enable type coercion Sub-task Open Unassigned  
          35.
          DataTypeExtractor does not check for default constructor Sub-task Closed Timo Walther  
          36.
          Support for comparing instance of all types Sub-task Open Unassigned  
          37.
          Update data type documentation for 1.11 Sub-task Closed Timo Walther  
          38.
          Remove RowDataTypeInfo Sub-task Closed Timo Walther  
          39.
          Update Table API set operations to the new type system Sub-task Closed Timo Walther  
          40.
          Calling ROW() in a UDF results in UnsupportedOperationException Sub-task Closed Timo Walther  
          41.
          Retain the column name when converting a Table to a DataStream Sub-task Open Unassigned  
          42.
          Type mismatch when using ROW in computed column Sub-task Open Unassigned  
          43.
          Cannot create empty array using ARRAY[] Sub-task Open Unassigned  
          44.
          Update TableResult.collect()/TableResult.print() to the new type system Sub-task Closed Timo Walther  
          45.
          Support TIMESTAMP WITH TIME ZONE Sub-task Open Unassigned  
          46.
          GenericTypeInfo equality issue Sub-task Open Unassigned  
          47.
          DataTypes implementation for flink-csv with ZonedDateTime support Sub-task Open Unassigned  
          48.
          Add a union type Sub-task Open Unassigned  
          49.
          Convert CharSequence to String when registering / converting a Table Sub-task Open Unassigned  
          50.
          Add native Avro type support to the Table API & SQL Sub-task Open Unassigned  
          51.
          The precision in document and code of `INTERVAL DAY(p1) TO SECOND(p2)` are inconsistent Sub-task Open Unassigned  

            Activity

              People

              • Assignee:
                twalthr Timo Walther
                Reporter:
                twalthr Timo Walther
              • Votes:
                0 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 9h
                  9h