Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-2894

WorkerSinkTask doesn't handle rewinding offsets on rebalance

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.9.0.0
    • Fix Version/s: 0.10.1.0
    • Component/s: KafkaConnect
    • Labels:
      None

      Description

      rewind() is only invoked at the beginning of each poll(). This means that if a rebalance occurs in the poll, it's feasible to get data that doesn't match a request to change offsets during the rebalance. I think the consumer will hold on to consumer data across the rebalance if it is reassigned the same offset, so there may already be data ready to be delivered. Additionally we may already have data in an incomplete messageBatch that should be discarded when the rewind is requested.

      While connectors that care about this (i.e. ones that manage their own offsets) can handle this correctly by tracking the offsets they're expecting to see, it's a hassle, error prone, an pretty unintuitive.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                liquanpei Liquan Pei
                Reporter:
                ewencp Ewen Cheslack-Postava
              • Votes:
                1 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: