Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-4942

Kafka Connect: Offset committing times out before expected

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.2.0
    • Fix Version/s: 0.11.0.0
    • Component/s: KafkaConnect
    • Labels:
      None

      Description

      On Kafka 0.10.2.0
      I run a connector that deals with a lot of data, in a kafka connect cluster

      When the offsets are getting committed, I get the following:

      [2017-03-23 03:56:25,134] INFO WorkerSinkTask{id=MyConnector-1} Committing offsets (org.apache.kafka.connect.runtime.WorkerSinkTask)
      [2017-03-23 03:56:25,135] WARN Commit of WorkerSinkTask{id=MyConnector-1} offsets timed out (org.apache.kafka.connect.runtime.WorkerSinkTask)
      

      If you look at the timestamps, they're 1 ms apart. My settings are the following:

      	offset.flush.interval.ms = 120000
      	offset.flush.timeout.ms = 60000
      	offset.storage.topic = _connect_offsets
      

      It seems the offset flush timeout setting is completely ignored for the look of the logs. I would expect the timeout message to happen 60 seconds after the commit offset INFO message, not 1 millisecond later.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              stephane.maarek@gmail.com Stephane Maarek
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: