Kafka
  1. Kafka
  2. KAFKA-919

Disabling of auto commit is ignored during consumer group rebalancing

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.0
    • Component/s: consumer
    • Labels:
      None
    • Environment:
      java 7/linux

      Description

      From the mailing list:

      In one of our applications using Kafka, we are using the high-level consumer to pull messages from our topic.

      Because we pull messages from topics in discrete units (e.g., an hour's worth of messages), we want to control explicitly when offsets are committed.

      Even though "auto.commit.enable" is set to false, during consumer group rebalancing, offsets are committed anyway, regardless of the setting of this flag.

      Is this a bug? Or just a known gap in offset management? I do see plenty of notes on the wiki suggesting future releases may enable applications using the high-level consumer to have more fine-grained control over offset management.

      I also fully realize that different applications have different needs, and meeting all of them with a clean API can be challenging.

      In the case of this application, the high-level consumer solves the problem of locating the correct in a cluster for a given topic, so there are advantages to using it, even if we are not using it to balance fetch load across multiple consumers. We ideally have only 1 consumer active per consumer group, and can tolerate some duplicate messages. But, the consumer groups make it easy for 1 consumer to recover at the correct starting point, should the prior consumer in the group have failed before doing a commit.

      1. kafka-919.patch
        0.7 kB
        Phil Hargett

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        2m 25s 1 Phil Hargett 24/May/13 17:40
        Patch Available Patch Available Open Open
        8s 1 Phil Hargett 24/May/13 17:40
        Open Open Resolved Resolved
        12h 21m 1 Jun Rao 25/May/13 06:02
        Resolved Resolved Closed Closed
        4d 12h 11m 1 Neha Narkhede 29/May/13 18:14
        Tony Stevenson made changes -
        Workflow Apache Kafka Workflow [ 13053186 ] no-reopen-closed, patch-avail [ 13055685 ]
        Tony Stevenson made changes -
        Workflow no-reopen-closed, patch-avail [ 12783869 ] Apache Kafka Workflow [ 13053186 ]
        Neha Narkhede made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Jun Rao made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Assignee Neha Narkhede [ nehanarkhede ]
        Fix Version/s 0.8 [ 12317244 ]
        Resolution Fixed [ 1 ]
        Hide
        Jun Rao added a comment -

        Thanks for the patch. Committed to 0.8.

        Show
        Jun Rao added a comment - Thanks for the patch. Committed to 0.8.
        Phil Hargett made changes -
        Attachment kafka-919.patch [ 12584709 ]
        Hide
        Phil Hargett added a comment -

        Very simple patch; no change to any comments

        Show
        Phil Hargett added a comment - Very simple patch; no change to any comments
        Phil Hargett made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Phil Hargett made changes -
        Field Original Value New Value
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Phil Hargett added a comment -

        Note from Jun on the users mailing list: "it's a bug."

        Show
        Phil Hargett added a comment - Note from Jun on the users mailing list: "it's a bug."
        Phil Hargett created issue -

          People

          • Assignee:
            Unassigned
            Reporter:
            Phil Hargett
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development