Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-19226

[Kinesis] [EFO] Connector reaches default max attempts for describeStream and describeStreamConsumer when parallelism is high

    XMLWordPrintableJSON

    Details

      Description

      Background

      When lazily registering the stream consumer on apps with high parallelism, EFO connector hits default maximum number of attempts when calling describeStream and describeStreamConsumer on the Kinesis Streams API.

      The default FullJitterBackoff constants are tuned to prevent this when parallelism of 1024 is used.

      Scope

      • See FLIP for full list of configuration options
      • Suggested changes:
        flink.stream.describe.maxretries 50
        flink.stream.describe.backoff.base 2000L
        flink.stream.describe.backoff.max 5000L
        flink.stream.describestreamconsumer.maxretries 50
        flink.stream.describestreamconsumer.backoff.base 2000L
        flink.stream.describestreamconsumer.backoff.max 5000L

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                hlteoh37 Hong Liang Teoh
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: