Kafka
  1. Kafka
  2. KAFKA-1030

Addition of partitions requires bouncing all the consumers of that topic

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.0
    • Component/s: None
    • Labels:
      None

      Description

      Consumer may not notice new partitions because the propagation of the metadata to servers can be delayed.

      Options:
      1. As Jun suggested on KAFKA-956, the easiest fix would be to read the new partition data from zookeeper instead of a kafka server.
      2. Run a fetch metadata loop in consumer, and set auto.offset.reset to smallest once the consumer has started.

      1 sounds easier to do. If 1 causes long delays in reading all partitions at the start of every rebalance, 2 may be worth considering.

      The same issue affects MirrorMaker when new topics are created, MirrorMaker may not notice all partitions of the new topics until the next rebalance.

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            Guozhang Wang
            Reporter:
            Swapnil Ghike
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development