Major Description
Kafka Streams uses "delete record" requests to aggressively purge data from repartition topics. Those request are sent each time we commit.
For at-least-once with a default commit interval of 30 seconds, this works fine. However, for exactly-once with a default commit interval of 100ms, it's very aggressive. The main issue is broker side, because the broker logs every "delete record" request, and thus broker logs are spammed if EOS is enabled.
We should consider to add a new config (eg `delete.record.interval.ms` or similar) to have a dedicated config for "delete record" requests, to decouple it from the commit interval config and allow to purge data less aggressively, even if the commit interval is small to avoid the broker side log spamming.
Attachments
Issue Links
- links to