Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-1030

Addition of partitions requires bouncing all the consumers of that topic

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.0
    • Component/s: None
    • Labels:
      None

      Description

      Consumer may not notice new partitions because the propagation of the metadata to servers can be delayed.

      Options:
      1. As Jun suggested on KAFKA-956, the easiest fix would be to read the new partition data from zookeeper instead of a kafka server.
      2. Run a fetch metadata loop in consumer, and set auto.offset.reset to smallest once the consumer has started.

      1 sounds easier to do. If 1 causes long delays in reading all partitions at the start of every rebalance, 2 may be worth considering.

      The same issue affects MirrorMaker when new topics are created, MirrorMaker may not notice all partitions of the new topics until the next rebalance.

        Attachments

          Activity

            People

            • Assignee:
              guozhang Guozhang Wang
              Reporter:
              swapnilghike Swapnil Ghike
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: