Kafka
  1. Kafka
  2. KAFKA-346

Don't call commitOffsets() during rebalance

    Details

    • Type: Improvement Improvement
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.7, 0.7.1, 0.8.0
    • Fix Version/s: None
    • Component/s: core
    • Labels:
      None

      Description

      A sample use-case

      If I read the source correctly then offsets can be committed at any
      time (whenever there is a change in consumer or broker zk registry).
      Our application doesn't use auto-commit in order to batch some
      messages together, process them and if everything went fine, we call
      commitOffsets(). If, for any reason, the processing of messages does
      not succeed, we rely on Kafka's promise to re-deliver the messages.

      But if ZKRebalancerListener triggers a rebalance before our "batch" of
      messages is full, then offsets will be committed even if the messages
      have not been processed yet by our application. So if then processing
      of these messages fails, we basically lost them, right?

      (See discussion at http://www.mail-archive.com/kafka-users@incubator.apache.org/msg01415.html)

      1. KAFKA-346.patch
        11 kB
        Peter Romianowski

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Unassigned
              Reporter:
              Peter Romianowski
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:

                Development