[KAFKA-1436] Idempotent Producer / Duplicate Detection

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 0.8.0, 0.8.1, 0.8.1.1, 0.8.2.0, 0.10.1.0
• Component/s: consumer, producer
• Assignee: Neha Narkhede
• Reporter: James Thornton
• Created: 05/May/14 03:26
• Updated: 10/Mar/17 10:07
• Resolved: 10/Mar/17 10:07

Description

Dealing with duplicate messages is one of the major issues for teams using Kafka, and Jay Kreps posted a page on implementing an Idempotent Producer to address this issue:

https://cwiki.apache.org/confluence/display/KAFKA/Idempotent+Producer

MapDB 1.0 (https://github.com/jankotek/MapDB) was just released, and either it or Java Chronicle (https://github.com/OpenHFT/Java-Chronicle/) could be embedded within each broker to provide a high-performance, random-access, off-heap store for request IDs.

As Jay points out in his post, global unique request IDs probably aren't needed, but if that need should arise, Twitter's Snowflake service (https://github.com/twitter/snowflake/) might be useful.

Issue Links

• is duplicated by: KAFKA-4817 Implement idempotent producer