[FLINK-36637] Fix ListShards for Kinesis streams with large number of shards

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: aws-connector-4.4.0
• Fix Version/s: aws-connector-5.0.0
• Component/s: Connectors / Kinesis
• Labels: pull-request-available
• Assignee: Elphas Toringepi
• Reporter: Elphas Toringepi
• Created: 31/Oct/24 10:40
• Updated: 04/Nov/24 22:12
• Resolved: 31/Oct/24 16:16

Description

The new Kinesis source fails to consume from streams with a large number of shards because it cannot successfully list the shards. This issue was discovered on a stream with 1024 shards.

The following exception is thrown:

org.apache.flink.util.FlinkException: Global failure triggered by OperatorCoordinator for 'Source: Kinesis Stream Source' (operator bc764cd8ddf7a0cff126f51c16239658).
    at org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder$LazyInitializedCoordinatorContext.failJob(OperatorCoordinatorHolder.java:651)
    ...
Caused by: org.apache.flink.connector.kinesis.source.exception.KinesisStreamsSourceException: Failed to list shards.
    at org.apache.flink.connector.kinesis.source.enumerator.KinesisStreamsSourceEnumerator.processDiscoveredSplits(KinesisStreamsSourceEnumerator.java:375)
    at org.apache.flink.runtime.source.coordinator.ExecutorNotifier.lambda$notifyReadyAsync$1(ExecutorNotifier.java:83)
    at org.apache.flink.util.ThrowableCatchingRunnable.run(ThrowableCatchingRunnable.java:40)
    ... 6 more
Caused by: org.apache.flink.connector.kinesis.source.exception.KinesisStreamsSourceException: Unable to list shards
    at org.apache.flink.connector.kinesis.source.enumerator.KinesisStreamsSourceEnumerator.initialDiscoverSplits(KinesisStreamsSourceEnumerator.java:260)
    at org.apache.flink.runtime.source.coordinator.ExecutorNotifier.lambda$notifyReadyAsync$2(ExecutorNotifier.java:80)
    ... 6 more
Caused by: org.apache.flink.kinesis.shaded.software.amazon.awssdk.services.kinesis.model.InvalidArgumentException: NextToken and StreamARN cannot be provided together. (Service: Kinesis, Status Code: 400, Request ID: d0ca3ae9-975e-6f46-8f56-70aa948cfa30, Extended Request ID: +4y3p3zgJ9MOZ7FOL4C1LnhrVm69QLZXbwvCyZ7f3bfArn66oKcYz5J0xAGEKMA7JBGa6wzI7aOTwdHbPofuhfOY+LpjTSUk)
    ...

 

 

Issue Links

• links to: GitHub Pull Request #178