Kafka
  1. Kafka
  2. KAFKA-242

Subsequent calls of ConsumerConnector.createMessageStreams cause Consumer offset to be incorrect

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.7
    • Fix Version/s: None
    • Component/s: clients
    • Labels:
      None

      Description

      When calling ConsumerConnector.createMessageStreams in rapid succession, the Consumer offset is incorrectly advanced causing the consumer to lose messages. This seems to happen when createMessageStreams is called before the rebalancing triggered by the previous call to createMessageStreams has completed.

      1. kafka.log
        7 kB
        David Arthur

        Activity

        Hide
        Stefan Miklosovic added a comment -

        Jay Kreps I am hitting the same issues

        I opened JIRA here https://issues.apache.org/jira/browse/KAFKA-2331

        I am calling ConsumerConnector.createMessageStreams fastly and it seems that it is not handling rebalancing of partitions correctly.

        Show
        Stefan Miklosovic added a comment - Jay Kreps I am hitting the same issues I opened JIRA here https://issues.apache.org/jira/browse/KAFKA-2331 I am calling ConsumerConnector.createMessageStreams fastly and it seems that it is not handling rebalancing of partitions correctly.
        Hide
        Jay Kreps added a comment -

        Does this bug still exist in 0.8?

        Show
        Jay Kreps added a comment - Does this bug still exist in 0.8?
        David Arthur made changes -
        Field Original Value New Value
        Attachment kafka.log [ 12509694 ]
        Hide
        David Arthur added a comment -

        Annotated log messages from two different test runs. For each test run, 20k messages were written to "mytopic" and "mytopic1" each with no consumers running. Once the messages had all been sent, the consumers were started up

        Show
        David Arthur added a comment - Annotated log messages from two different test runs. For each test run, 20k messages were written to "mytopic" and "mytopic1" each with no consumers running. Once the messages had all been sent, the consumers were started up
        David Arthur created issue -

          People

          • Assignee:
            Unassigned
            Reporter:
            David Arthur
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:

              Development