Kafka
  1. Kafka
  2. KAFKA-1030

Addition of partitions requires bouncing all the consumers of that topic

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.0
    • Component/s: None
    • Labels:
      None

      Description

      Consumer may not notice new partitions because the propagation of the metadata to servers can be delayed.

      Options:
      1. As Jun suggested on KAFKA-956, the easiest fix would be to read the new partition data from zookeeper instead of a kafka server.
      2. Run a fetch metadata loop in consumer, and set auto.offset.reset to smallest once the consumer has started.

      1 sounds easier to do. If 1 causes long delays in reading all partitions at the start of every rebalance, 2 may be worth considering.

      The same issue affects MirrorMaker when new topics are created, MirrorMaker may not notice all partitions of the new topics until the next rebalance.

        Activity

          People

          • Assignee:
            Guozhang Wang
            Reporter:
            Swapnil Ghike
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development