Details
-
New Feature
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
Kafka should have an emergency retention setting for max disk used to prevent the broker running out of disk and partitions going off line. When this max is reached, Kafka could perhaps delete segments from the largest topics. Would have to be used with care as current behaviour is to preserve data at the cost of availability. This would favour availability over data retention.
At the moment it's quite hard to reason about disk usage and Kafka as the max byte settings are all per partition, and the math can get complicated when you have lots of topics of different use cases and sizes..