[KAFKA-18032] ProducerFencedExceptions should wipe out the state store only for the tasks that are related to the failed transaction

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.9.0
• Component/s: streams
• Assignee: Eduwer Camacaro
• Reporter: Eduwer Camacaro
• Created: 18/Nov/24 16:19
• Updated: 18/Nov/24 16:19

Description

A ProducerFencedException is thrown when the StreamThread reaches the `transaction.timeout.ms` limit and EOS is enabled, causing dirty closes on the running StreamTask in order to protect Exactly Once Semantics. Even if the failed transaction contained records associated with a single stream task, it wipes out all associated StreamTask states. In the inflight transaction, the StreamsProducer should keep track of the associated task and only close 'dirty' these tasks when the producer is fenced.

Steps to reproduce:

1. Deploys a Kafka Streams instance that uses multiple tasks in a single StreamThread
2. Set the `transaction.timeout.ms` small enough to cause a timeout during processing.
3. Send load to your input topics.

 

Check how many of your StreamTasks are currently closed dirty.

stream-thread [app1-StreamThread-1] task [0_0] Closed dirty