Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-7760

Add broker configuration to set minimum value for segment.bytes and segment.ms

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      If someone set segment.bytes or segment.ms at topic level to a very small value (e.g. segment.bytes=1000 or segment.ms=1000), Kafka will generate a very high number of segment files. This can bring down the whole broker due to hitting the maximum open file (for log) or maximum number of mmap-ed file (for index).

      To prevent that from happening, I would like to suggest adding two new items to the broker configuration:

      • min.topic.segment.bytes, defaults to 1048576: The minimum value for segment.bytes. When someone sets topic configuration segment.bytes to a value lower than this, Kafka throws an error INVALID VALUE.
      • min.topic.segment.ms, defaults to 3600000: The minimum value for segment.ms. When someone sets topic configuration segment.ms to a value lower than this, Kafka throws an error INVALID VALUE.

      Thanks

      Badai

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                dulvinw Dulvin Witharane
                Reporter:
                badai Badai Aqrandista
              • Votes:
                1 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated: