[KAFKA-7108] "Exactly-once" stream breaks production exception handler contract

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.1.0
• Component/s: streams
• Labels: exactly-once
• Assignee: Unassigned
• Reporter: Anna O
• Created: 27/Jun/18 06:35
• Updated: 30/Nov/21 17:58

Description

I have a stream configured with "default.production.exception.handler" that is supposed to log the error and continue. When I set "processing.guarantee" to "exactly_once" it appeared that retryable NotEnoughReplicasException that passed the production exception handler was rethrown by the TransactionManager wrapped with KafkaException and terminated the stream thread:

org.apache.kafka.common.KafkaException: Cannot execute transactional method because we are in an error stateat org.apache.kafka.clients.producer.internals.TransactionManager.maybeFailWithError(TransactionManager.java:784) ~[kafka-clients-1.1.0.jar:?]
at org.apache.kafka.clients.producer.internals.TransactionManager.sendOffsetsToTransaction(TransactionManager.java:250) ~[kafka-clients-1.1.0.jar:?]
at org.apache.kafka.clients.producer.KafkaProducer.sendOffsetsToTransaction(KafkaProducer.java:617) ~[kafka-clients-1.1.0.jar:?]
at org.apache.kafka.streams.processor.internals.StreamTask.commitOffsets(StreamTask.java:357) ~[kafka-streams-1.1.0.jar:?]
at org.apache.kafka.streams.processor.internals.StreamTask.access$000(StreamTask.java:53) ~[kafka-streams-1.1.0.jar:?]
at org.apache.kafka.streams.processor.internals.StreamTask$1.run(StreamTask.java:316) ~[kafka-streams-1.1.0.jar:?]
at org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:208) ~[kafka-streams-1.1.0.jar:?]
at org.apache.kafka.streams.processor.internals.StreamTask.commit(StreamTask.java:307) ~[kafka-streams-1.1.0.jar:?]
at org.apache.kafka.streams.processor.internals.StreamTask.commit(StreamTask.java:297) ~[kafka-streams-1.1.0.jar:?]
at org.apache.kafka.streams.processor.internals.AssignedTasks$1.apply(AssignedTasks.java:67) ~[kafka-streams-1.1.0.jar:?]
at org.apache.kafka.streams.processor.internals.AssignedTasks.applyToRunningTasks(AssignedTasks.java:357) [kafka-streams-1.1.0.jar:?]
at org.apache.kafka.streams.processor.internals.AssignedTasks.commit(AssignedTasks.java:347) [kafka-streams-1.1.0.jar:?]
at org.apache.kafka.streams.processor.internals.TaskManager.commitAll(TaskManager.java:403) [kafka-streams-1.1.0.jar:?]
at org.apache.kafka.streams.processor.internals.StreamThread.maybeCommit(StreamThread.java:994) [kafka-streams-1.1.0.jar:?]
at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:811) [kafka-streams-1.1.0.jar:?]
at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:750) [kafka-streams-1.1.0.jar:?]
at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:720) [kafka-streams-1.1.0.jar:?]
Caused by: org.apache.kafka.common.errors.NotEnoughReplicasException: Messages are rejected since there are fewer in-sync replicas than required.

 

Issue Links

• is related to: KAFKA-10829 Kafka Streams handle produce exception improvement