Kafka
  1. Kafka
  2. KAFKA-167

Move partition assignment to the broker

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Currently partitions are assigned to consumers for consumption via a co-ordination algorithm by the consumers. This means there is effectively no master broker node, which is nice in a way, but makes the consumer logic complex to implement. It would be good to move the co-ordination to the brokers. This would make implementation of a consumer client much much easier, since only minimal zk interaction would be required. There are a number of details that would have to be worked out to finalize the protocol. Discussion is here:

      http://mail-archives.apache.org/mod_mbox/incubator-kafka-dev/201109.mbox/%3cCAFbh0Q2ADjXcbCYqGh8TB4jXBAs3Hq5RLxk9o9w76we-bWanVw@mail.gmail.com%3e

        Issue Links

          Activity

          Hide
          nicu marasoiu added a comment -

          Hi,

          So I would suggest this: for each consumer group, one broker can be elected as partition/replica assignment HA coordinator using a standard election mechanism on zookeeper already available in Curator.

          I think it is nicer to distribute as much as possible, and, while recognizing that the algorithm is more deterministic, stable and convergent when using a centralized coordinator, there is no reason to pick the same coordinator for different consumer groups.

          Apart from the mail discussion thread, can you please indicate any discussion on design, implementation, so that I can try to pick a small part and code something for it?

          Thanks,
          Nicu Marasoiu

          Show
          nicu marasoiu added a comment - Hi, So I would suggest this: for each consumer group, one broker can be elected as partition/replica assignment HA coordinator using a standard election mechanism on zookeeper already available in Curator. I think it is nicer to distribute as much as possible, and, while recognizing that the algorithm is more deterministic, stable and convergent when using a centralized coordinator, there is no reason to pick the same coordinator for different consumer groups. Apart from the mail discussion thread, can you please indicate any discussion on design, implementation, so that I can try to pick a small part and code something for it? Thanks, Nicu Marasoiu
          Hide
          Jay Kreps added a comment -

          This is being done (finally) in the new consumer implementation.

          Show
          Jay Kreps added a comment - This is being done (finally) in the new consumer implementation.

            People

            • Assignee:
              Unassigned
              Reporter:
              Jay Kreps
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development