[KAFKA-228] Reduce duplicate messages served by the kafka consumer for uncompressed topics

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.7
• Fix Version/s: 0.7.1
• Assignee: Neha Narkhede
• Reporter: Neha Narkhede
• Created: 17/Dec/11 18:39
• Updated: 07/Jan/12 00:11
• Resolved: 07/Jan/12 00:11

Description

Kafka guarantees at-least once delivery of messages.The high level consumer provides highly available partitioned consumption of data within the same consumer group. In the event of broker failures or consumer failures within a group, the high level consumer rebalances and redistributes the topic partitions evenly amongst the consumers in a group. With the current design, during this rebalancing operation, Kafka introduces duplicates in the consumed data.

This JIRA improves the rebalancing operation and the consumer iterator design to guarantee 0 duplicates while consuming uncompressed topics. There will be a small number of duplicates while serving compressed data, but it will be bound by the compression batch size.

Issue Links

• duplicates: KAFKA-198 Avoid duplicated message during consumer rebalance