Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-10897

kafka quota optimization

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Blocker
    • Resolution: Unresolved
    • Affects Version/s: 2.7.0
    • Fix Version/s: None
    • Component/s: admin, clients, config, consumer, core
    • Labels:
      None

      Description

      1.The current quota dimensions is as follows:

      /config/users/<user>/clients/<client-id>
      /config/users/<user>/clients/<default>
      /config/users/<user>
      /config/users/<default>/clients/<client-id>
      /config/users/<default>/clients/<default>
      /config/users/<default>
      /config/clients/<client-id>
      /config/clients/<default>

      2. Existing problems:

       

      2.1.The quota dimensions is not fine enough.
      2.2.When multiple users on the same broker produce and consume a large amount of data at the same time, if you want the broker to run normally, you must make the sum of all user quota byte not exceed the upper throughput limit of the broker.
      2.3.Even if all the user rate does not reach the upper limit of the broker, but all the user rate is concentrated on a few disks and exceeds the read-write load of the disk, all the produce and consume requests will be blocked.
      2.4.Sometimes it's just one topic rate increase sharply under the user, so we just need to limit the increase sharply topics.
      

       

      3. Suggestions for improvement

      3.1. Add the upper limit of single broker quota byte.
      3.2. Add the upper limit of single disk quota byte on the broker.
      3.3. Add topic quota dimensions.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              1012293987 yangyijun
              Reviewer:
              Angel Alvarez Pascua
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: