[KAFKA-6038] Repartition topics could be much more transient

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: streams
• Labels: optimization
• Assignee: Unassigned
• Reporter: Guozhang Wang
• Created: 10/Oct/17 00:52
• Updated: 29/Nov/17 18:58
• Resolved: 29/Nov/17 18:58

Description

Unlike changelog topics, the repartition topics could just be short-lived than eating up the storage space on Kafka brokers. Today users have different ways to configure them with short retention such as enforce a retention of 30 minutes with small log segment sizes, or use AppendTime for repartition topics. All these would be cumbersome and Streams should just do this automatically.

One way to do it is use the “purgeData” admin API (KIP-107) such that after the offset of the input topics are committed, if the input topics are actually repartition topics, we would purge the data immediately. One tricky thing to consider though, is upon (re-)starting the application, if the repartition topics are used for restoring the states, we need to re-fill these topics in the right way in order for restoration purposes, and there might be some devils in the implementation details.

Issue Links

• duplicates: KAFKA-6150 Make Repartition Topics Transient