[FLINK-35541] Introduce retry limiting for AWS connector sinks

• Type: Technical Debt
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: aws-connector-4.2.0, aws-connector-4.3.0
• Component/s: Connectors / AWS, Connectors / DynamoDB, Connectors / Firehose, Connectors / Kinesis
• Assignee: Unassigned
• Reporter: Aleksandr Pilipenko
• Created: 06/Jun/24 09:17
• Updated: 11/Jun/24 12:54

Description

Currently if the record write operation in the sink consistently fails with retriable error, sinks will retry indefinitely. In case when cause of the error is not resolved this may lead to poison pill.

 

Proposal here is to add a configurable retry limit for each record. Users can specify a maximum retry per record, and the sink will fail once the retry limit is reached.

 

We will implement this for all AWS connectors:

• DDBSink
• Firehose Sink
• Kinesis Sink