[STORM-2104] New Kafka spout crashes if partitions are reassigned while tuples are in-flight

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0, 1.1.0
• Fix Version/s: 2.0.0, 1.1.0, 1.0.4
• Component/s: storm-kafka-client
• Epic Link: Release Apache Storm 1.1.0
• Assignee: Stig Rohde Døssing
• Reporter: Stig Rohde Døssing
• Created: 20/Sep/16 14:46
• Updated: 14/Jul/17 20:56
• Resolved: 29/Dec/16 00:34

Description

The new KafkaSpout may throw NPEs if partitions are reassigned while tuples are in-flight. The ack function assumes that the spout instance is always responsible for tuples it emitted, which isn't true if partitions were reassigned since the tuple was emitted. The fail function also assumes that failed tuples should be replayed, which is useless if the tuple is for a partition the spout isn't assigned, since it then can't commit the tuple if it succeeds. Both functions should check that the spout instance is responsible for the incoming tuple before scheduling it for retry or adding it to the acked list.

Issue Links

• duplicates: STORM-2180 KafkaSpout ack() should handle re-balance
• is duplicated by: STORM-2207 Kafka Spout NullPointerException during ack
• links to: GitHub Pull Request #1696
• links to: GitHub Pull Request #1818