[FLINK-31772] KinesisStreamsSink Performance regression due to AIMD rate limiting strategy

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.16.1, aws-connector-4.1.0
• Fix Version/s: aws-connector-4.2.0
• Component/s: Connectors / Kinesis
• Labels: pull-request-available
• Assignee: Ahmed Hamdy
• Reporter: Ahmed Hamdy
• Created: 11/Apr/23 12:07
• Updated: 16/May/23 13:43
• Resolved: 16/May/23 13:43

Description

Issue

While benchmarking the KinesisStreamSink for 1.15 against the legacy FlinkKinesisProduced , it is observed that the new sink has a performance regression against the deprecated sink for same environment setting.

Further investigation identified that the AIMD Ratelimiting strategy is the bottleneck for the regression.

Attached results for KinesisStreamSink against

{FlinkKinesisProducer}

and

{KinesisStreamSink}

after disabling AIMDRatelimitingStrategy

Environment Settings

• Benchmarking was performed on AWS KDA.
• Application logic is just sending records downstream
• Application parallelism was tested to be 1.
• Kinesis stream number of shards was tested with 8 and 12.
• payload size was 1Kb and 100Kb.

Issue Links

• links to: GitHub Pull Request #70