[FLINK-24905] [FLIP-171] KDS implementation of Async Sink Table API

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.15.0
• Component/s: Connectors / Kinesis
• Labels: pull-request-available
• Assignee: Ahmed Hamdy
• Reporter: Zichen Liu
• Created: 15/Nov/21 14:54
• Updated: 14/Feb/22 13:10
• Resolved: 01/Feb/22 16:04

Description

Motivation

User stories:
As a Flink user, I’d like to use the Table API for the new Kinesis Data Streams  sink.

Scope:

• Introduce AsyncDynamicTableSink that enables Sinking Tables into Async Implementations.
• Implement a new KinesisDynamicTableSink that uses KinesisDataStreamSink Async Implementation and implements AsyncDynamicTableSink.
• The implementation introduces Async Sink configurations as optional options in the table definition, with default values derived from the KinesisDataStream default values.
• Unit/Integration testing. modify KinesisTableAPI tests for the new implementation, add unit tests for AsyncDynamicTableSink and KinesisDynamicTableSink and KinesisDynamicTableSinkFactory.
• Java / code-level docs.

References

More details to be found https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink

 

Issue Links

• causes: FLINK-25792 Async Sink Base is too being flushed too frequently resulting in backpressure even when buffer is near empty
• Dependent: FLINK-24041 [FLIP-171] Generic AsyncSinkBase
• is blocked by: FLINK-24234 [FLIP-171] Byte Based & Time Based Flushing for AsyncSinkBase
• is blocked by: FLINK-24227 [FLIP-171] KDS implementation of Async Sink
• is cloned by: FLINK-26056 CLONE - [FLIP-171] KDS implementation of Async Sink Table API
• links to: GitHub Pull Request #17907
• links to: GitHub Pull Request #18348
• links to: GitHub Pull Request #18468

Sub-Tasks

• [FLIP-171] Add SQL Client Uber jar for KinesisDataStreams Sink.: Closed