[STORM-1869] Add ability for Kafka Spout to do limited retries on failed tuples, and redirect down a "failed" stream

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: storm-kafka
• Assignee: Unassigned
• Reporter: Stephen Powis
• Created: 29/May/16 23:46
• Updated: 30/May/16 05:26

Description

Summary

KafkaSpout provides no way to limit the number of times a tuple can fail within your topology. I think it'd be great to allow configuring a hard limit to the number of times a tuple can be failed before it is no longer retried.

Additionally, it'd be fantastic if the spout could do a one-time emit of this tuple down a "failed" stream. This stream would allow you to handle these failed tuples in some way – persist the failed tuple to some other storage system to be replayed later? logged? ignored? tons of things you could possibly want to do with these.