[STORM-2546] Kafka spout can stall / get stuck due to edge case with failing tuples

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0, 1.1.0
• Fix Version/s: 2.0.0, 1.2.0
• Component/s: storm-kafka-client
• Labels: pull-request-available
• Epic Link: Release Apache Storm 1.2.0
• Assignee: Stig Rohde Døssing
• Reporter: Prasanna Ranganathan
• Created: 08/Jun/17 07:48
• Updated: 16/Nov/17 06:13
• Resolved: 16/Nov/17 06:12

Description

The mechanism for replaying a failed tuple involves seeking the kafka consumer to the failing offset and then re-emitting it into the topology. A tuple, when emitted the first time, will have an entry created in OffsetManager. This entry will be removed only after the tuple is successfully acknowledged and its offset successfully committed. Till then, commits for offsets beyond the failing offset for that TopicPartition will be blocked.

It is possible that when the spout seeks the consumer to the failing offset, the corresponding kafka message is not returned in the poll response. This can happen due to that offset being deleted or compacted away. In this scenario that partition will be blocked from committing and progressing.

Issue Links

• relates to: STORM-2430 Potential Race condition in Kafka Spout
• links to: GitHub Pull Request #2307
• links to: GitHub Pull Request #2423