[FLINK-26890] DynamoDB consumer error consuming partitions close to retention

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.16.0, 1.15.2
• Fix Version/s: 1.17.0, 1.15.3, 1.16.1
• Component/s: Connectors / Kinesis
• Labels: pull-request-available
• Assignee: Elphas Toringepi
• Reporter: Danny Cranmer
• Created: 28/Mar/22 15:13
• Updated: 04/Nov/22 12:33
• Resolved: 04/Nov/22 12:33

Description

Background

The Amazon Kinesis Data Streams consumer supports consuming from Amazon DynamoDB via the DynamoDB Streams Kinesis Adapter.

Problem

We have seen instances of consumer throwing ResouceNotFoundException when attempting to invoke GetShardIterator.

com.amazonaws.services.kinesis.model.ResourceNotFoundException: Requested resource not found: Shard does not exist 

According to the DynamoDB team, the DescribeStream call may return shard IDs that are no longer valid, and this exception needs to be handled by the client.

Solution

Modify the DynamoDB consumer to treat ResourceNotFoundException as a shard closed signal.

Issue Links

• links to: GitHub Pull Request #21102
• links to: GitHub Pull Request #21237
• links to: GitHub Pull Request #21238