[STORM-2296] Kafka spout - no duplicates on topic leader changes

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.2
• Fix Version/s: 2.0.0, 1.1.0, 1.0.4
• Component/s: storm-kafka
• Epic Link: Release Apache Storm 1.1.0
• Assignee: Ernestas Vaiciukevičius
• Reporter: Ernestas Vaiciukevičius
• Created: 17/Jan/17 07:26
• Updated: 19/Jun/18 14:51
• Resolved: 03/Feb/17 23:28

Description

Current behavior of Kafka spout emits duplicate tuples whenever Kafka topic leader's change.
In case of exception caused by leader changes, PartitionManagers are simply recreated losing the state about which tuples were already emitted and new PartitionManager re-emits them again.

This is fine as at-least-once is fulfilled, but still it would be better to not emit duplicate data if possible.
Moreover this could be easily avoided by moving the state related to emitted tuples from old PartitionManager to new one.

Pull requests implementing this:
1.0.x-branch - https://github.com/apache/storm/pull/1873
1.x-branch - https://github.com/apache/storm/pull/1888

Pull request for related bugfix: https://github.com/apache/storm/pull/1940

Issue Links

• is related to: STORM-2361 Kafka spout - after topic leader change, it stops committing offsets to ZK
• relates to: STORM-3090 The same offset value is used by the same partition number of different topics.
• links to: GitHub Pull Request #1873
• links to: GitHub Pull Request #1888
• links to: GitHub Pull Request #1940