[FLINK-36270] DDB Streams Connector performance issue due to splitsAvailableForAssignment function

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: aws-connector-5.0.0
• Component/s: Connectors / DynamoDB
• Labels: pull-request-available
• Assignee: Abhi Gupta
• Reporter: Abhi Gupta
• Created: 12/Sep/24 12:08
• Updated: 04/Nov/24 22:11
• Resolved: 13/Sep/24 11:07

Description

In DDB Streams connector, while testing we found out that when we are spending a lot of time in markAsFinished function because we are calling splitsAvailableForAssignment which is O(N), and given n shards can be marked as finished concurrently, the algorithm becomes O(n^2). Change the algo to assign only child shards when a parent is finished. We can start tracking child shards of a shard in SplitTracker

Issue Links

• links to: GitHub Pull Request #164