[FLINK-20629] [Kinesis][EFO] Migrate from DescribeStream to DescribeStreamSummary

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.12.0
• Fix Version/s: 1.12.1, 1.13.0
• Component/s: Connectors / Kinesis
• Labels: pull-request-available
• Assignee: Danny Cranmer
• Reporter: Danny Cranmer
• Created: 16/Dec/20 13:24
• Updated: 21/Dec/20 06:33
• Resolved: 21/Dec/20 06:33

Description

Background
The Kinesis EFO connector invokes DescribeStream during startup to acquire the stream ARN. This call also includes the shard information and has a TPS of 10. A similar service exists, DescribeStreamSummary that has a TPS of 20 and a lighter response payload size.

During startup sources with high parallelism compete to call this service (in LAZY mode), resulting in backoff and retry. Essentially the startup time can grow by 1s for every 10 parallelism, due to the 10 TPS. Migrating to DescribeStreamSummary will improve startup time.

Scope
Migrate call to DescribeStream to use DescribeStreamSummary instead.

Note
I have targeted 1.12.1, let me know if we should instead target 1.13

Issue Links

• links to: GitHub Pull Request #14406