Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-346

Don't call commitOffsets() during rebalance

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.7, 0.7.1, 0.8.0
    • Fix Version/s: None
    • Component/s: core
    • Labels:
      None

      Description

      A sample use-case

      If I read the source correctly then offsets can be committed at any
      time (whenever there is a change in consumer or broker zk registry).
      Our application doesn't use auto-commit in order to batch some
      messages together, process them and if everything went fine, we call
      commitOffsets(). If, for any reason, the processing of messages does
      not succeed, we rely on Kafka's promise to re-deliver the messages.

      But if ZKRebalancerListener triggers a rebalance before our "batch" of
      messages is full, then offsets will be committed even if the messages
      have not been processed yet by our application. So if then processing
      of these messages fails, we basically lost them, right?

      (See discussion at http://www.mail-archive.com/kafka-users@incubator.apache.org/msg01415.html)

        Attachments

        1. KAFKA-346.patch
          11 kB
          Peter Romianowski

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                peter.romianowski Peter Romianowski
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: