[STORM-2077] KafkaSpout doesn't retry failed tuples

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.0.2
• Component/s: storm-kafka-client
• Assignee: Unassigned
• Reporter: Tobias Maier
• Created: 01/Sep/16 11:11
• Updated: 19/Feb/18 19:36
• Resolved: 30/Jun/17 22:23

Description

KafkaSpout does not retry all failed tuples.

We used following Configuration:
Map<String, Object> props = new HashMap<>();
props.put(KafkaSpoutConfig.Consumer.GROUP_ID, "c1");
props.put(KafkaSpoutConfig.Consumer.KEY_DESERIALIZER, ByteArrayDeserializer.class.getName());
props.put(KafkaSpoutConfig.Consumer.VALUE_DESERIALIZER, ByteArrayDeserializer.class.getName());
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, broker.bootstrapServer());

KafkaSpoutStreams kafkaSpoutStreams = new KafkaSpoutStreams.Builder(FIELDS_KAFKA_EVENT, new String[]

{"test-topic"}

).build();

KafkaSpoutTuplesBuilder<byte[], byte[]> kafkaSpoutTuplesBuilder = new KafkaSpoutTuplesBuilder.Builder<>(new KeyValueKafkaSpoutTupleBuilder("test-topic")).build();
KafkaSpoutRetryService retryService = new KafkaSpoutLoggedRetryExponentialBackoff(KafkaSpoutLoggedRetryExponentialBackoff.TimeInterval.milliSeconds(1), KafkaSpoutLoggedRetryExponentialBackoff.TimeInterval.milliSeconds(1), 3, KafkaSpoutLoggedRetryExponentialBackoff.TimeInterval.seconds(1));

KafkaSpoutConfig<byte[], byte[]> config = new KafkaSpoutConfig.Builder<>(props, kafkaSpoutStreams, kafkaSpoutTuplesBuilder, retryService)
.setFirstPollOffsetStrategy(UNCOMMITTED_LATEST)
.setMaxUncommittedOffsets(30)
.setOffsetCommitPeriodMs(10)
.setMaxRetries(3)
.build();

kafkaSpout = new org.apache.storm.kafka.spout.KafkaSpout<>(config);

The downstream bolt fails every tuple and we expect, that those tuple will all be replayed. But that's not the case for every tuple.

Issue Links

• duplicates: STORM-2087 Storm-kafka-client: Failed tuples are not always replayed
• duplicates: STORM-2229 KafkaSpout does not resend failed tuples