Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-14039

Flink Kinesis consumer: configurable per-shard consumption rate when running in adaptive mode

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Connectors / Kinesis
    • Labels:
      None

      Description

      Currently, Flink kinesis connector has a fixed 2MB target rate (per-shard) when running in adaptive rate mode.  In specific scenarios, it is desirable that users would want a different target rate. For example, when two Kinesis consumers share a common stream, the user may want to de-prioritize one stream such that it runs with a target rate < 2MB. 

      It is relatively straightforward to implement this feature – simply add a per-shard target rate consumer config and has the default set to 2MB. 

       

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              yxu-apache Ying Xu
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: