Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-5445

flink kafka consumer should support rate control

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Duplicate
    • 1.1.3
    • 1.8.0
    • Connectors / Kafka
    • None

    Description

      Now kafka connector does not support rate control at all, this may lead checkpoints to fail and expire if downstream processor is slow. Though this will not compromise the guarantee of flink's checkpoint, this will stop kafka consumer from comitting offsets since kafka consumer commits offsets in the callback of checkpoint complete. The committed offsets is important for monitoring purpose. So I think we need to support rate limit control in kafka consumer.

      Attachments

        Issue Links

          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. FLINK-11501 Add a ratelimiting feature to the FlinkKafkaConsumer

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              liurenjie1024 Renjie Liu
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: