Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-6535

Set default retention ms for Streams repartition topics to Long.MAX_VALUE

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0
    • Component/s: streams
    • Labels:

      Description

      After KIP-220 / KIP-204, repartition topics in Streams are transient, so it is better to set its default retention to infinity to allow any records be pushed to it with old timestamps (think: bootstrapping, re-processing) and just rely on the purging API to keeping its storage small.

      More specifically, in RepartitionTopicConfig we have a few default overrides for repartition topic configs, we should just add the override for TopicConfig.RETENTION_MS_CONFIG to set it to Long.MAX_VALUE. This still allows users to override themselves if they want via StreamsConfig.TOPIC_PREFIX. We need to add unit test to verify this update takes effect.

      In addition to the code change, we also need to have doc changes in streams/upgrade_guide.html specifying this default value change.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Khairy Khaireddine Rezgui
                Reporter:
                guozhang Guozhang Wang
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: