Kafka
  1. Kafka
  2. KAFKA-1000

Inbuilt consumer offset management feature for kakfa

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 0.8.1
    • Fix Version/s: 0.8.2.0
    • Component/s: consumer
    • Labels:

      Description

      Kafka currently stores offsets in zookeeper. This is a problem for several reasons. First it means the consumer must embed the zookeeper client which is not available in all languages. Secondly offset commits are actually quite frequent and Zookeeper does not scale this kind of high-write load.

      This Jira is for tracking the phase #2 of Offset Management [0]. Joel and I have been working on this. [1] is the overall design of the feature.

      [0] : https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management
      [1] : https://cwiki.apache.org/confluence/display/KAFKA/Inbuilt+Consumer+Offset+Management

        Activity

        Tejas Patil created issue -
        Tejas Patil made changes -
        Field Original Value New Value
        Description This Jitra corresponds to the phase #2 of Offset Management [0]. After KAFKA-657 [1], we have a protocol for consumers to commit and fetch offsets from brokers. Currently, consumers are not using this API and directly talking with Zookeeper.

        This Jira will involve following:
        1. Consumers should now use the offset commit API
        2. Add a special topic in kafka for storing offsets
        3. Pluggable offset commit logic so that users can pick the existing ZK based storage or inbuilt storage for offsets.
        4. Leader brokers would now maintain an additional hash table of offsets for the group-topic-partitions that they lead
        5. Consumers send their offset fetch requests to brokers. Receiving broker may or may not be the leader for the requested group-topic-partitions so would have to forward the request to respective leader brokers.

        Joel and I have been working on this and I would share the initial patch and a writeup covering the nitty-gritty of the implementation.

        [0] : https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management
        [1] : https://issues.apache.org/jira/browse/KAFKA-657
        This Jira is for tracking the phase #2 of Offset Management [0]. After KAFKA-657 [1], we have a protocol for consumers to commit and fetch offsets from brokers. Currently, consumers are not using this API and directly talking with Zookeeper.
        Joel and I have been working on this.

        [0] : https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management
        [1] : https://issues.apache.org/jira/browse/KAFKA-657
        Tejas Patil made changes -
        Summary Inbuilt consumer offset management Inbuilt consumer offset management feature for kakfa
        Tejas Patil made changes -
        Description This Jira is for tracking the phase #2 of Offset Management [0]. After KAFKA-657 [1], we have a protocol for consumers to commit and fetch offsets from brokers. Currently, consumers are not using this API and directly talking with Zookeeper.
        Joel and I have been working on this.

        [0] : https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management
        [1] : https://issues.apache.org/jira/browse/KAFKA-657
        This Jira is for tracking the phase #2 of Offset Management [0]. Joel and I have been working on this. [1] is the overall design of the feature.

        [0] : https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management
        [1] : https://cwiki.apache.org/confluence/display/KAFKA/Inbuilt+Consumer+Offset+Management
        Jay Kreps made changes -
        Description This Jira is for tracking the phase #2 of Offset Management [0]. Joel and I have been working on this. [1] is the overall design of the feature.

        [0] : https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management
        [1] : https://cwiki.apache.org/confluence/display/KAFKA/Inbuilt+Consumer+Offset+Management
        Kafka currently stores offsets in zookeeper. This is a problem for several reasons. First it means the consumer must embed the zookeeper client which is not available in all languages. Secondly offset commits are actually quite frequent and Zookeeper does not scale this kind of high-write load.

        This Jira is for tracking the phase #2 of Offset Management [0]. Joel and I have been working on this. [1] is the overall design of the feature.

        [0] : https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management
        [1] : https://cwiki.apache.org/confluence/display/KAFKA/Inbuilt+Consumer+Offset+Management
        Manikumar Reddy made changes -
        Fix Version/s 0.8.2 [ 12326167 ]

          People

          • Assignee:
            Tejas Patil
            Reporter:
            Tejas Patil
          • Votes:
            5 Vote for this issue
            Watchers:
            14 Start watching this issue

            Dates

            • Created:
              Updated:

              Development