Yan Fang, I thought about this a little bit more,
I was thinking as a short term solution:
What if the users can set the TTL to same value as the Kafka's retention time ? Joshua Hartman Will that work for your use case ? Kafka will wipe out data with a fixed time, within that time interval it's OK to restore your store. This means that we'll revert the exception patch and just allow users to create Changelog streams for even TTL based ones. I agree, it's kind of a gamble, where you will see users accidentally enabling TTL and do not understand the implications of using it. Or even worse, keep writing more and more data till they run of disk before they realize that kafka doesn't expunge expired records. But it's too hard to control. We can do further optimizations in another ticket - to create changelog topic with TTL retention, or if already exists, verify if it matches. But that's the best we can do. What are your thoughts ?
As long term solution:
We store some meta information with each key-value (preferably in the key - but this will cause log compaction to fail), so we need to think more about this in detail, but essentially a meta information as a part of the message itself. And we do a restore, we skip the ones which have expired and write a expired_key -> null, which will trigger log compaction and wipe the data out eventually. This probably needs a lot more work and thought.