Kafka
  1. Kafka
  2. KAFKA-198

Avoid duplicated message during consumer rebalance

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: 0.7
    • Fix Version/s: 0.7.1
    • Component/s: None
    • Labels:
      None

      Description

      Currently, a consumer can get duplicated messages when a rebalance is triggered. It would be good if we can eliminate those duplicated messages.

        Issue Links

          Activity

          Hide
          Jun Rao added a comment -

          This is a bit tricky to get right. The following is one possible design:
          1. In ConsumerIterator, add a method clearCurrentChunk(). This method will clear the current chunk being iterated. This method has to by synchronized with makeNext().

          2. In KafaMessageStream, add a method clear() which calls consumerIterator.clearCurrentChunk

          3. In Fetcher, break initConnections() into 2 methods: stopConnections() and startConnections().

          4. In ZookeeperConsumerConnector.updateFetcher. Do the following:
          a. fetcher.stopConnections
          b. for each new Fetcher to be created
          b1. clear fetcher queue
          b2. call KafkaMessageStream.clear
          c. call commitOffsets
          d. fetcher.startConnections

          Show
          Jun Rao added a comment - This is a bit tricky to get right. The following is one possible design: 1. In ConsumerIterator, add a method clearCurrentChunk(). This method will clear the current chunk being iterated. This method has to by synchronized with makeNext(). 2. In KafaMessageStream, add a method clear() which calls consumerIterator.clearCurrentChunk 3. In Fetcher, break initConnections() into 2 methods: stopConnections() and startConnections(). 4. In ZookeeperConsumerConnector.updateFetcher. Do the following: a. fetcher.stopConnections b. for each new Fetcher to be created b1. clear fetcher queue b2. call KafkaMessageStream.clear c. call commitOffsets d. fetcher.startConnections
          Hide
          Joe Stein added a comment -

          KAFKA-228 resolve this same issue

          Show
          Joe Stein added a comment - KAFKA-228 resolve this same issue

            People

            • Assignee:
              Neha Narkhede
              Reporter:
              Jun Rao
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development