Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-6038

Repartition topics could be much more transient

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: streams
    • Labels:

      Description

      Unlike changelog topics, the repartition topics could just be short-lived than eating up the storage space on Kafka brokers. Today users have different ways to configure them with short retention such as enforce a retention of 30 minutes with small log segment sizes, or use AppendTime for repartition topics. All these would be cumbersome and Streams should just do this automatically.

      One way to do it is use the “purgeData” admin API (KIP-107) such that after the offset of the input topics are committed, if the input topics are actually repartition topics, we would purge the data immediately. One tricky thing to consider though, is upon (re-)starting the application, if the repartition topics are used for restoring the states, we need to re-fill these topics in the right way in order for restoration purposes, and there might be some devils in the implementation details.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                guozhang Guozhang Wang
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: