Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-4702

Kafka consumer must commit offsets asynchronously

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 1.1.2
    • 1.1.3, 1.2.0
    • Connectors / Kafka
    • None

    Description

      The offset commit calls to Kafka may occasionally take very long.
      In that case, the notifyCheckpointComplete() method blocks for long and the KafkaConsumer cannot make progress and cannot perform checkpoints.

      Kafka 0.9+ have methods to commit asynchronously.
      We should use those and make sure no more than one commit is concurrently in progress, to that commit requests do not pile up.

      Attachments

        Activity

          People

            sewen Stephan Ewen
            sewen Stephan Ewen
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: