[STORM-2087] Storm-kafka-client: Failed tuples are not always replayed

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0, 1.0.2
• Fix Version/s: 2.0.0, 1.1.0
• Component/s: storm-kafka-client
• Labels: pull-request-available
• Epic Link: Release Apache Storm 1.1.0
• Assignee: Jeffrey Ryan Fenchel
• Reporter: Jeffrey Ryan Fenchel
• Created: 11/Sep/16 02:45
• Updated: 19/Feb/18 19:36
• Resolved: 20/Dec/16 07:47

Description

I am working with kafka 10 and the storm-kafka-client from master. It appears that tuples are not always being replayed when they are failed.

With a topology that randomly fails tuples a small percentage of the time I found that the committed kafka offset would get stuck and eventually processing would stop even though the committed offset was no where near the end of the topic.

I have also replicated the issue in unit tests with this PR:
https://github.com/apache/storm/pull/1679

It seems that increasing the number of times I call nextTuple for the in order case will make it work, but it doesn't seem to help the case where tuples are failed out of order from which they were emitted.

Issue Links

• is duplicated by: STORM-2077 KafkaSpout doesn't retry failed tuples
• is duplicated by: STORM-2229 KafkaSpout does not resend failed tuples
• links to: GitHub Pull Request #1679
• links to: GitHub Pull Request #1826