Details
-
Improvement
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
0.10.1.0
Description
Currently state store replication always go through a compact kafka topic. For some state stores, e.g. JoinWindow, there are no duplicates in the store, there is not much benefit using a compacted topic.
The problem of using compacted topic is the records can stay in kafka broker forever. In my use case, my key is ad_id, it's incrementing all the time, not bounded, I am worried the disk space on broker for that topic will go forever.
I think we either need the capability to purge the compacted records on broker, or allow us to specify different compact option for state store replication.
Attachments
Issue Links
- links to
