[FLINK-34339] Add connector abstraction layer to remove reliance on AWS SDK classes

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Connectors / Kinesis
• Assignee: Unassigned
• Reporter: Danny Cranmer
• Created: 02/Feb/24 09:24
• Updated: 02/Feb/24 10:13

Description

In order to shim DDB streams we need to be able to support the Stream/Shard/Record etc concepts without tying to a specific implementation. This will allow us to mimic the KDS/DDB streams support in the old connector by providing a shim at the AWS SDK client.

1. Model software.amazon.awssdk.services.kinesis classes as native concepts
2. Push down any usage of software.amazon.awssdk.services.kinesis to a KDS specific class
3. Ensure that the bulk of the connector logic is reusable, the top level class would be implementation specific and shim in the write client factories and configuration

Issue Links

• blocks: FLINK-31922 Port over Kinesis Client configurations for retry and backoff
• blocks: FLINK-31980 Implement support for EFO in new KDS Source
• blocks: FLINK-32324 Implement watermark alignment on KDS source
• blocks: FLINK-33180 Implement support for processing subset of shards/traffic
• blocks: FLINK-34340 Add support for DDB Streams for DataStream API
• is blocked by: FLINK-24438 Port Kinesis Source to new Source API (FLIP-27)