[FLINK-19976] FLIP-136: Improve interoperability between DataStream and Table API

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.14.0
• Component/s: Table SQL / API
• Assignee: Unassigned
• Reporter: Timo Walther
• Created: 04/Nov/20 15:07
• Updated: 07/Sep/21 07:31
• Resolved: 07/Sep/21 07:31

Description

The Table API has received many new features over the last year. It supports a new type system (FLIP-37), connectors support changelogs (FLIP-95), we have well defined internal data structures (FLIP-95), support for result retrieval in an interactive fashion (FLIP-84), and soon new TableDescriptors (FLIP-129).

However, the interfaces from and to DataStream API have not been touched during the introduction of these new features and are kind of outdated. The interfaces lack important functionality that is available in Table API but not exposed to DataStream API users. DataStream API is still our most important API which is why a good interoperability is crucial.

This FLIP is a mixture of different topics that improve the interoperability between DataStream and Table API in terms of:

• DataStream ↔ Table conversion
• translation of type systems TypeInformation ↔ DataType
• schema definition (incl. rowtime, watermarks, primary key)
• changelog handling
• row handling in DataStream API

See:
https://cwiki.apache.org/confluence/pages/resumedraft.action?draftId=158872537&draftShareId=69731f11-580f-4bfe-847f-e5eb3e585f1f&

Issue Links

• contains: FLINK-8545 Implement upsert stream table source
• mentioned in: Page Loading...