[FLINK-11184] Rework TableSource and TableSink interfaces

• Type: New Feature
• Status: Open
• Priority: Not a Priority
• Resolution: Unresolved
• Component/s: Table SQL / API
• Labels: auto-deprioritized-major auto-deprioritized-minor auto-unassigned
• Assignee: Unassigned
• Reporter: Timo Walther
• Created: 17/Dec/18 11:37
• Updated: 25/Dec/21 22:38

Description

There are a couple of shortcomings with the current TableSource and TableSink interface design. Some of the issues are covered in a basic design document that was published a while ago.

The design document has not been updated for some time and partially overlaps with the current SQL DDL discussion for the CREATE TABLE statement on the ML.

What needs to be solved:

• How to unify sources and sinks in regards of schema and time attributes?
• How to define watermarks, timestamp extractors or timestamp ingestion?
• How to define primary keys and partitioning keys?
• How to differentiate between update modes for tables (i.e. how to read from a append, retraction, or update table)?
• How to express all of the above without pulling in to many dependencies on other Flink modules if source and sink interfaces are located in flink-table-spi package?

As of the current state of the discussion, it seems that we might extend TableSchema to allow for returning the information above and remove current interfaces such as DefinedRowtimeAttribute or DefinedFieldMapping.

Issue Links

• blocks: FLINK-10688 Remove flink-table dependencies from flink-connectors