[KAFKA-1012] Implement an Offset Manager and hook offset requests to it

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.8.2.0
• Component/s: consumer
• Assignee: Tejas Patil
• Reporter: Tejas Patil
• Created: 19/Aug/13 19:40
• Updated: 06/Jan/15 06:32
• Resolved: 09/Apr/14 18:57

Description

After KAFKA-657, we have a protocol for consumers to commit and fetch offsets from brokers. Currently, consumers are not using this API and directly talking with Zookeeper.

This Jira will involve following:
1. Add a special topic in kafka for storing offsets
2. Add an OffsetManager interface which would handle storing, accessing, loading and maintaining consumer offsets
3. Implement offset managers for both of these 2 choices : existing ZK based storage or inbuilt storage for offsets.
4. Leader brokers would now maintain an additional hash table of offsets for the group-topic-partitions that they lead
5. Consumers should now use the OffsetCommit and OffsetFetch API