Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-4702

Kafka consumer must commit offsets asynchronously

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.1.2
    • Fix Version/s: 1.2.0, 1.1.3
    • Component/s: Kafka Connector
    • Labels:
      None

      Description

      The offset commit calls to Kafka may occasionally take very long.
      In that case, the notifyCheckpointComplete() method blocks for long and the KafkaConsumer cannot make progress and cannot perform checkpoints.

      Kafka 0.9+ have methods to commit asynchronously.
      We should use those and make sure no more than one commit is concurrently in progress, to that commit requests do not pile up.

        Attachments

          Activity

            People

            • Assignee:
              StephanEwen Stephan Ewen
              Reporter:
              StephanEwen Stephan Ewen
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: