[KAFKA-1000] Inbuilt consumer offset management feature for kakfa

• Type: New Feature
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.8.1
• Fix Version/s: 0.8.2.0
• Component/s: consumer
• Labels: features
• Assignee: Tejas Patil
• Reporter: Tejas Patil
• Created: 05/Aug/13 16:13
• Updated: 16/Jun/15 18:23
• Resolved: 16/Jun/15 18:23

Description

Kafka currently stores offsets in zookeeper. This is a problem for several reasons. First it means the consumer must embed the zookeeper client which is not available in all languages. Secondly offset commits are actually quite frequent and Zookeeper does not scale this kind of high-write load.

This Jira is for tracking the phase #2 of Offset Management [0]. Joel and I have been working on this. [1] is the overall design of the feature.

[0] : https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management
[1] : https://cwiki.apache.org/confluence/display/KAFKA/Inbuilt+Consumer+Offset+Management

Sub-Tasks

• Junits and system tests for inbuilt offset management: Open