Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-20629

[Kinesis][EFO] Migrate from DescribeStream to DescribeStreamSummary

    XMLWordPrintableJSON

    Details

      Description

      Background
      The Kinesis EFO connector invokes DescribeStream during startup to acquire the stream ARN. This call also includes the shard information and has a TPS of 10. A similar service exists, DescribeStreamSummary that has a TPS of 20 and a lighter response payload size.

      During startup sources with high parallelism compete to call this service (in LAZY mode), resulting in backoff and retry. Essentially the startup time can grow by 1s for every 10 parallelism, due to the 10 TPS. Migrating to DescribeStreamSummary will improve startup time.

      Scope
      Migrate call to DescribeStream to use DescribeStreamSummary instead.

      Note
      I have targeted 1.12.1, let me know if we should instead target 1.13

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                danny.cranmer Danny Cranmer
                Reporter:
                danny.cranmer Danny Cranmer
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: