[FLINK-33260] Custom Error Handling for Kinesis EFO Consumer

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: aws-connector-4.2.0
• Fix Version/s: aws-connector-4.3.0
• Component/s: Connectors / Kinesis
• Labels: pull-request-available
• Assignee: Emre Kartoglu
• Reporter: Danny Cranmer
• Created: 12/Oct/23 18:18
• Updated: 04/Jan/24 13:39
• Resolved: 04/Jan/24 13:39

Description

Background

The Kinesis Consumer exposes various configuration that allows the user to define retry and backoff strategies when dealing with errors. However, the configuration does not allow the user to configure which errors are retryable, or different strategies for different errors. The error handling logic is hard coded within the connector. Over time we discover errors that should be retryable that are not, for example KDS throwing 500 on SubscribeToShare or transient DNS issues. When these arise the user can either fork-fix the connector or log an issue and wait for the next version.

Scope

Add the ability for the user to define retry/backoff strategy per error. This could be achieved using flexible configuration keys, or allowing the user to register their own retry strategies on the connector

 

Issue Links

• is related to: FLINK-33991 Custom Error Handling for Kinesis Polling Consumer
• links to: GitHub Pull Request #110