[KAFKA-7760] Add broker configuration to set minimum value for segment.bytes and segment.ms

• Type: Improvement
• Status: Reopened
• Priority: Major
• Resolution: Unresolved
• Labels: kip newbie
• Assignee: Badai Aqrandista
• Reporter: Badai Aqrandista
• Created: 20/Dec/18 00:06
• Updated: 12/Jul/21 21:12

Description

If someone set segment.bytes or segment.ms at topic level to a very small value (e.g. segment.bytes=1000 or segment.ms=1000), Kafka will generate a very high number of segment files. This can bring down the whole broker due to hitting the maximum open file (for log) or maximum number of mmap-ed file (for index).

To prevent that from happening, I would like to suggest adding two new items to the broker configuration:

• min.topic.segment.bytes, defaults to 1048576: The minimum value for segment.bytes. When someone sets topic configuration segment.bytes to a value lower than this, Kafka throws an error INVALID VALUE.
• min.topic.segment.ms, defaults to 3600000: The minimum value for segment.ms. When someone sets topic configuration segment.ms to a value lower than this, Kafka throws an error INVALID VALUE.

Thanks

Badai

Issue Links

• duplicates: KAFKA-13040 Increase minimum value of segment.ms and segment.bytes
• is cloned by: KAFKA-7761 CLONE - Add broker configuration to set minimum value for segment.bytes and segment.ms
• is duplicated by: KAFKA-7761 CLONE - Add broker configuration to set minimum value for segment.bytes and segment.ms
• links to: GitHub Pull Request #6911
• links to: GitHub Pull Request #6920
• links to: GitHub Pull Request #6921
• links to: GitHub Pull Request #6923
• links to: KIP-760: Increase minimum value of segment.ms and segment.bytes