Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-19976

FLIP-136: Improve interoperability between DataStream and Table API

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 1.14.0
    • Table SQL / API
    • None

    Description

      The Table API has received many new features over the last year. It supports a new type system (FLIP-37), connectors support changelogs (FLIP-95), we have well defined internal data structures (FLIP-95), support for result retrieval in an interactive fashion (FLIP-84), and soon new TableDescriptors (FLIP-129).

      However, the interfaces from and to DataStream API have not been touched during the introduction of these new features and are kind of outdated. The interfaces lack important functionality that is available in Table API but not exposed to DataStream API users. DataStream API is still our most important API which is why a good interoperability is crucial.

      This FLIP is a mixture of different topics that improve the interoperability between DataStream and Table API in terms of:

      • DataStream ↔ Table conversion
      • translation of type systems TypeInformation ↔ DataType
      • schema definition (incl. rowtime, watermarks, primary key)
      • changelog handling
      • row handling in DataStream API

      See:
      https://cwiki.apache.org/confluence/pages/resumedraft.action?draftId=158872537&draftShareId=69731f11-580f-4bfe-847f-e5eb3e585f1f&

      Attachments

        Issue Links

          1.
          Support all conversion classes in Row.equals/hashCode Sub-task Closed Timo Walther
          2.
          Improve dealing with Row in DataStream API Sub-task Closed Timo Walther
          3.
          Add a converter from TypeInformation to DataType Sub-task Closed Timo Walther
          4.
          Allow SQL expressions in Table API Sub-task Closed Timo Walther
          5.
          Replace TableSchema with dedicated CatalogSchema Sub-task Closed Timo Walther
          6.
          Create a utility to create DataStream API's DataType and Schema Sub-task Closed Timo Walther
          7.
          Add new StreamTableEnvironment.fromDataStream Sub-task Closed Timo Walther
          8.
          Add new StreamTableEnvironment.toDataStream Sub-task Closed Timo Walther
          9.
          Allow casting between row and structured type Sub-task Closed Timo Walther
          10.
          Add StreamTableEnvironment.to/fromChangelogStream Sub-task Closed Timo Walther
          11.
          Make structured type's fields more lenient during casting Sub-task Closed Timo Walther
          12.
          Add Scala implicit conversions for new API methods Sub-task Closed Timo Walther
          13.
          Propagate Boundedness of SourceFunctionProvider to Transformation Sub-task Closed Timo Walther
          14.
          Disable AutoWatermarkInterval for bounded legacy sources Sub-task Closed Timo Walther
          15.
          Disable AllVerticesInSameSlotSharingGroupByDefault in DataStream batch mode Sub-task Closed Timo Walther
          16.
          Disable key sorting for Transformations created within table planner Sub-task Closed Timo Walther
          17.
          Support object reuse disabled in OperatorChain Sub-task Closed Timo Walther
          18.
          Expose a consistent GlobalDataExchangeMode Sub-task Closed Timo Walther
          19.
          Simplify BlinkExecutorFactory stack Sub-task Closed Timo Walther
          20.
          Use a layered configuration in Executor and Planner Sub-task Closed Timo Walther
          21.
          Use pipeline name consistently across DataStream API and Table API Sub-task Closed Timo Walther
          22.
          Use consistent managed memory weights for StreamNode Sub-task Closed Timo Walther
          23.
          Support DataStream batch mode in StreamTableEnvironment Sub-task Closed Timo Walther
          24.
          Add StreamStatementSet.attachAsDataStream Sub-task Closed Timo Walther
          25.
          Test StreamTableEnvironment batch mode in Python Sub-task Closed Dian Fu
          26.
          Deprecate toAppendStream and toRetractStream Sub-task Closed Timo Walther
          27.
          Type mismatch when declaring SOURCE_WATERMARK on TIMESTAMP_LTZ column Sub-task Closed Timo Walther
          28.
          Add documentation how to interact with DataStream API Sub-task Closed Timo Walther
          29.
          Align FLIP-136 (Improve interoperability between DataStream and Table API) in PyFlink Table API Sub-task Resolved Huang Xingbo
          30.
          Add documentation for batch mode in StreamTableEnvironment Sub-task Closed Timo Walther

          Activity

            People

              Unassigned Unassigned
              twalthr Timo Walther
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: