Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-242

Subsequent calls of ConsumerConnector.createMessageStreams cause Consumer offset to be incorrect

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Auto Closed
    • Affects Version/s: 0.7
    • Fix Version/s: None
    • Component/s: clients
    • Labels:
      None

      Description

      When calling ConsumerConnector.createMessageStreams in rapid succession, the Consumer offset is incorrectly advanced causing the consumer to lose messages. This seems to happen when createMessageStreams is called before the rebalancing triggered by the previous call to createMessageStreams has completed.

      1. kafka.log
        7 kB
        David Arthur

        Activity

        Hide
        mumrah David Arthur added a comment -

        Annotated log messages from two different test runs. For each test run, 20k messages were written to "mytopic" and "mytopic1" each with no consumers running. Once the messages had all been sent, the consumers were started up

        Show
        mumrah David Arthur added a comment - Annotated log messages from two different test runs. For each test run, 20k messages were written to "mytopic" and "mytopic1" each with no consumers running. Once the messages had all been sent, the consumers were started up
        Hide
        jkreps Jay Kreps added a comment -

        Does this bug still exist in 0.8?

        Show
        jkreps Jay Kreps added a comment - Does this bug still exist in 0.8?
        Hide
        smiklosovic Stefan Miklosovic added a comment -

        Jay Kreps I am hitting the same issues

        I opened JIRA here https://issues.apache.org/jira/browse/KAFKA-2331

        I am calling ConsumerConnector.createMessageStreams fastly and it seems that it is not handling rebalancing of partitions correctly.

        Show
        smiklosovic Stefan Miklosovic added a comment - Jay Kreps I am hitting the same issues I opened JIRA here https://issues.apache.org/jira/browse/KAFKA-2331 I am calling ConsumerConnector.createMessageStreams fastly and it seems that it is not handling rebalancing of partitions correctly.
        Hide
        jeffwidman Jeff Widman added a comment -

        Does this bug still exist in 0.10?

        Show
        jeffwidman Jeff Widman added a comment - Does this bug still exist in 0.10?
        Hide
        z88897050 ylzhang added a comment -
        Show
        z88897050 ylzhang added a comment - can upgrade and use new consumer, you can get it from https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Detailed+Consumer+Coordinator+Design
        Hide
        omkreddy Manikumar added a comment -

        Closing inactive issue. The old consumer is no longer supported.

        Show
        omkreddy Manikumar added a comment - Closing inactive issue. The old consumer is no longer supported.

          People

          • Assignee:
            Unassigned
            Reporter:
            mumrah David Arthur
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development