[KAFKA-1000] Inbuilt consumer offset management feature for kakfa

• Type: New Feature
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.8.1
• Fix Version/s: 0.8.2.0
• Component/s: consumer
• Labels: features
• Assignee: Tejas Patil
• Reporter: Tejas Patil
• Created: 05/Aug/13 16:13
• Updated: 16/Jun/15 18:23
• Resolved: 16/Jun/15 18:23

Description

Kafka currently stores offsets in zookeeper. This is a problem for several reasons. First it means the consumer must embed the zookeeper client which is not available in all languages. Secondly offset commits are actually quite frequent and Zookeeper does not scale this kind of high-write load.

This Jira is for tracking the phase #2 of Offset Management [0]. Joel and I have been working on this. [1] is the overall design of the feature.

[0] : https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management
[1] : https://cwiki.apache.org/confluence/display/KAFKA/Inbuilt+Consumer+Offset+Management

Sub-Tasks

• Implement an Offset Manager and hook offset requests to it: Resolved
• Modify existing tools as per the changes in KAFKA-1000: Resolved
• Junits and system tests for inbuilt offset management: Open
• documentation for inbuilt offset management: Resolved