[FLINK-17688] Support consuming Kinesis' enhanced fanout for flink-connector-kinesis

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.10.1
• Fix Version/s: 1.12.0
• Component/s: Connectors / Kinesis
• Assignee: Danny Cranmer
• Reporter: roland
• Created: 14/May/20 10:40
• Updated: 14/Jan/21 09:26
• Resolved: 21/Sep/20 08:35

Description

AWS Kinesis enhanced fanout is a feature that allows consumers to consume a separated sub-stream without competing with other consumers. (https://docs.aws.amazon.com/streams/latest/dev/building-enhanced-consumers-api.html)

Yet, currently flink-connector-kinesis can only consume main streams, which may lead to tense competition.

A support for the enhanced fanout feature is quite useful for AWS users.

Sub-Tasks

• [Kinesis][EFO] Test coverage improvements for FlinkKinesisConsumer: Closed
• [Kinesis][EFO] Introduce RecordPublisher Interface: Resolved
• [Kinesis][EFO] Add AWS SDK v2.x dependency and KinesisProxyV2: Closed
• [Kinesis][EFO] FanOutRecordPublisher and EFO support: Closed
• [Kinesis][EFO] Configuration deserialisation and validation: Closed
• [Kinesis][EFO] Stream consumer Registration/Deregistration: Closed
• [Kinesis][EFO] Update Kinesis Consumer website to reflect EFO: Closed
• CLONE - [Kinesis][EFO] Add AWS SDK v2.x dependency and KinesisProxyV2: Closed