Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-6365

Adapt default values of the Kinesis connector

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.2.1, 1.3.1, 1.4.0
    • Fix Version/s: 1.4.0, 1.3.2
    • Component/s: Kinesis Connector
    • Labels:
      None
    • Release Note:
      Hide
      Some default values for configurations for AWS API call behaviors in the Flink Kinesis Consumer was adapted for better default consumption performance: 1) SHARD_GETRECORDS_MAX default changed to 10,000, and 2) SHARD_GETRECORDS_INTERVAL_MILLIS default changed to 200ms.
      Show
      Some default values for configurations for AWS API call behaviors in the Flink Kinesis Consumer was adapted for better default consumption performance: 1) SHARD_GETRECORDS_MAX default changed to 10,000, and 2) SHARD_GETRECORDS_INTERVAL_MILLIS default changed to 200ms.

      Description

      As discussed in http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kinesis-connector-SHARD-GETRECORDS-MAX-default-value-td12332.html, it seems reasonable to change the default values of the Kinesis connector to follow KCL’s default settings. I suggest to adapt at least the values for SHARD_GETRECORDS_MAX and SHARD_GETRECORDS_INTERVAL_MILLIS.

      As a Kinesis shard is currently limited to 5 get operations per second, you can observe high ReadProvisionedThroughputExceeded rates with the current default value for SHARD_GETRECORDS_INTERVAL_MILLIS of 0; it seem reasonable to increase it to 200. As it's described in the email thread, it seems furthermore desirable to increase the default value for SHARD_GETRECORDS_MAX to 10000.

      The values that are used by the KCL can be found here: https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/KinesisClientLibConfiguration.java

      Thanks for looking into this!

      Steffen

        Attachments

          Activity

            People

            • Assignee:
              phoenixjiangnan Bowen Li
              Reporter:
              sthm Steffen Hausmann
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: