[KAFKA-12475] Kafka Streams breaks EOS with remote state stores - ASF JIRA

Attach files

Attach Screenshot

Add vote

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Open
Priority: Major
Resolution: Unresolved
Affects Version/s: None
Fix Version/s: None
Component/s: streams
Labels:
- needs-kip

Description

Currently in Kafka Streams, exactly-once semantics (EOS) require that the state stores be completely erased and restored from the changelog from scratch in case of an error. This erasure is implemented by closing the state store and then simply wiping out the local state directory. This works fine for the two store implementations provided OOTB, in-memory and rocksdb, but fails when the application includes a custom StateStore based on remote storage, such as Redis. In this case Streams will fail to erase any of the data before reinserting data from the changelog, resulting in possible duplicates and breaking the guarantee of EOS.