[FLINK-19226] [Kinesis] [EFO] Connector reaches default max attempts for describeStream and describeStreamConsumer when parallelism is high

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Abandoned
• Fix Version/s: 1.12.0
• Component/s: Connectors / Kinesis
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Hong Liang Teoh
• Created: 14/Sep/20 23:38
• Updated: 15/Sep/20 07:38
• Resolved: 15/Sep/20 07:27

Description

Background

When lazily registering the stream consumer on apps with high parallelism, EFO connector hits default maximum number of attempts when calling describeStream and describeStreamConsumer on the Kinesis Streams API.

The default FullJitterBackoff constants are tuned to prevent this when parallelism of 1024 is used.

Scope

• See FLIP for full list of configuration options
• Suggested changes:

  flink.stream.describe.maxretries	50
  flink.stream.describe.backoff.base	2000L
  flink.stream.describe.backoff.max	5000L
  flink.stream.describestreamconsumer.maxretries	50
  flink.stream.describestreamconsumer.backoff.base	2000L
  flink.stream.describestreamconsumer.backoff.max	5000L

Issue Links

• links to: GitHub Pull Request #13386