Description
Currently in Kafka Streams, exactly-once semantics (EOS) require that the state stores be completely erased and restored from the changelog from scratch in case of an error. This erasure is implemented by closing the state store and then simply wiping out the local state directory. This works fine for the two store implementations provided OOTB, in-memory and rocksdb, but fails when the application includes a custom StateStore based on remote storage, such as Redis. In this case Streams will fail to erase any of the data before reinserting data from the changelog, resulting in possible duplicates and breaking the guarantee of EOS.
Attachments
Issue Links
- is related to
-
KAFKA-12549 Allow state stores to opt-in transactional support
-
- Resolved
-