[STORM-2538] New kafka spout emits duplicate tuples

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Won't Fix
• Affects Version/s: 2.0.0, 1.x
• Component/s: storm-kafka-client
• Assignee: Hugo Da Cruz Louro
• Reporter: Parag Shah
• Created: 30/May/17 16:12
• Updated: 18/Jul/17 02:07
• Resolved: 18/Jul/17 02:07

Description

Currently, KafkaSpout in storm-kafka-client can cause duplicate tuples to be emitted. Reason is the implementation of ConsumerRebalanceListener interface is called by kafka everytime a new executor comes up. However, on PartitionsRevoked we already have some in flight tuples and are emitting the same ones from the new executor on which the onPartitionsAssigned was called. We need to make sure that we emit only one tuple per kafka message.

Issue Links

• is superceded by: STORM-2542 Deprecate storm-kafka-client KafkaConsumer.subscribe API subscriptions on 1.x and remove them as options in 2.x
• links to: GitHub Pull Request #2147