Kafka
  1. Kafka
  2. KAFKA-264

Change the consumer side load balancing and distributed co-ordination to use a consumer co-ordinator

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.7, 0.8.0
    • Fix Version/s: 0.9.0
    • Component/s: core
    • Labels:
      None

      Description

      A high level design for the zookeeper consumer is here - https://cwiki.apache.org/confluence/display/KAFKA/Consumer+co-ordinator

      1. KAFKA-264.v1.patch
        354 kB
        Guozhang Wang

        Issue Links

          Activity

          Hide
          Prashanth Menon added a comment - - edited

          Interesting stuff here. Just a question that logically follows from having a coordinator: what happens if the coordinating process goes down or becomes unavailable?

          Show
          Prashanth Menon added a comment - - edited Interesting stuff here. Just a question that logically follows from having a coordinator: what happens if the coordinating process goes down or becomes unavailable?
          Hide
          Neha Narkhede added a comment -

          The consumer co-ordinator itself is highly available, since it is elected from amongst the available consumer processes in a group. That can be done using the standard recipe for leader election using Zookeeper

          Show
          Neha Narkhede added a comment - The consumer co-ordinator itself is highly available, since it is elected from amongst the available consumer processes in a group. That can be done using the standard recipe for leader election using Zookeeper
          Hide
          Prashanth Menon added a comment -

          Hmm, forgive me if my questions are trivial

          What I mean is, the election of the leader can be handled by ZK, but what happens if the thread/process/machine with the coordinator goes down? Does every consumer group listen on the consumer-leader path in ZK and issue a new round of coordinator election, ala how it'll be done for replication leaders?

          Show
          Prashanth Menon added a comment - Hmm, forgive me if my questions are trivial What I mean is, the election of the leader can be handled by ZK, but what happens if the thread/process/machine with the coordinator goes down? Does every consumer group listen on the consumer-leader path in ZK and issue a new round of coordinator election, ala how it'll be done for replication leaders?
          Hide
          Neha Narkhede added a comment -

          If a process goes down, the ephemeral node for /consumers/[group]/leader will get deleted from Zookeeper. Each consumer process listens on that path, and triggers a leader election. However, these are very low level details of the implementation, which I intended to chalk out after the high level design looked good.

          You can read more about Zookeeper based leader election here - http://zookeeper.apache.org/doc/trunk/recipes.html#sc_leaderElection

          Show
          Neha Narkhede added a comment - If a process goes down, the ephemeral node for /consumers/ [group] /leader will get deleted from Zookeeper. Each consumer process listens on that path, and triggers a leader election. However, these are very low level details of the implementation, which I intended to chalk out after the high level design looked good. You can read more about Zookeeper based leader election here - http://zookeeper.apache.org/doc/trunk/recipes.html#sc_leaderElection
          Hide
          Prashanth Menon added a comment -

          Yup, got it, makes sense. I misinterpreted the wiki and thought election was only done once and at startup, hence my queston. I'll probably take another look when I get some spare time Thanks, Neha.

          Show
          Prashanth Menon added a comment - Yup, got it, makes sense. I misinterpreted the wiki and thought election was only done once and at startup, hence my queston. I'll probably take another look when I get some spare time Thanks, Neha.
          Hide
          Guozhang Wang added a comment -
          Show
          Guozhang Wang added a comment - Please refer to this wiki page for the detailed description of the implementation: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Detailed+Consumer+Coordinator+Design#KafkaDetailedConsumerCoordinatorDesign-11.Implementation
          Hide
          Scott Carey added a comment -

          Is this still going into 0.8? Is it already in there?

          Show
          Scott Carey added a comment - Is this still going into 0.8? Is it already in there?
          Hide
          Neha Narkhede added a comment -

          This is scheduled for 0.9

          Show
          Neha Narkhede added a comment - This is scheduled for 0.9
          Hide
          Scott Carey added a comment -

          I found it browsing "fix version=0.8"

          Show
          Scott Carey added a comment - I found it browsing "fix version=0.8"

            People

            • Assignee:
              Neha Narkhede
              Reporter:
              Neha Narkhede
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:

                Time Tracking

                Estimated:
                Original Estimate - 840h
                840h
                Remaining:
                Remaining Estimate - 840h
                840h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development