Description
After KIP-220 / KIP-204, repartition topics in Streams are transient, so it is better to set its default retention to infinity to allow any records be pushed to it with old timestamps (think: bootstrapping, re-processing) and just rely on the purging API to keeping its storage small.
More specifically, in RepartitionTopicConfig we have a few default overrides for repartition topic configs, we should just add the override for TopicConfig.RETENTION_MS_CONFIG to set it to Long.MAX_VALUE. This still allows users to override themselves if they want via StreamsConfig.TOPIC_PREFIX. We need to add unit test to verify this update takes effect.
In addition to the code change, we also need to have doc changes in streams/upgrade_guide.html specifying this default value change.
Attachments
Issue Links
- is duplicated by
-
KAFKA-7506 KafkaStreams repartition topic settings not suitable for processing old records
-
- Resolved
-
- links to
- mentioned in
-
Page Loading...