Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-242

Subsequent calls of ConsumerConnector.createMessageStreams cause Consumer offset to be incorrect

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Auto Closed
    • 0.7
    • None
    • clients
    • None

    Description

      When calling ConsumerConnector.createMessageStreams in rapid succession, the Consumer offset is incorrectly advanced causing the consumer to lose messages. This seems to happen when createMessageStreams is called before the rebalancing triggered by the previous call to createMessageStreams has completed.

      Attachments

        1. kafka.log
          7 kB
          David Arthur

        Activity

          mumrah David Arthur added a comment -

          Annotated log messages from two different test runs. For each test run, 20k messages were written to "mytopic" and "mytopic1" each with no consumers running. Once the messages had all been sent, the consumers were started up

          mumrah David Arthur added a comment - Annotated log messages from two different test runs. For each test run, 20k messages were written to "mytopic" and "mytopic1" each with no consumers running. Once the messages had all been sent, the consumers were started up
          jkreps Jay Kreps added a comment -

          Does this bug still exist in 0.8?

          jkreps Jay Kreps added a comment - Does this bug still exist in 0.8?

          jkreps I am hitting the same issues

          I opened JIRA here https://issues.apache.org/jira/browse/KAFKA-2331

          I am calling ConsumerConnector.createMessageStreams fastly and it seems that it is not handling rebalancing of partitions correctly.

          smiklosovic Stefan Miklosovic added a comment - jkreps I am hitting the same issues I opened JIRA here https://issues.apache.org/jira/browse/KAFKA-2331 I am calling ConsumerConnector.createMessageStreams fastly and it seems that it is not handling rebalancing of partitions correctly.
          jeffwidman Jeff Widman added a comment -

          Does this bug still exist in 0.10?

          jeffwidman Jeff Widman added a comment - Does this bug still exist in 0.10?
          z88897050 ylzhang added a comment - can upgrade and use new consumer, you can get it from https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Detailed+Consumer+Coordinator+Design
          omkreddy Manikumar added a comment -

          Closing inactive issue. The old consumer is no longer supported.

          omkreddy Manikumar added a comment - Closing inactive issue. The old consumer is no longer supported.

          People

            Unassigned Unassigned
            mumrah David Arthur
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: