Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-8293

Messages undelivered when small quotas applied

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.1.1
    • Fix Version/s: None
    • Component/s: core
    • Labels:

      Description

      I observe a strange Kafka behavior when using small quotas.

      For ex. I set quotas for the consumer like

       

      kafka-configs --zookeeper zookeeper:2181 --entity-type users --entity-name kafka --alter --add-config 'producer_byte_rate=2048000, consumer_byte_rate=256'

       

      If I send a small batch of messages as

       

      kafka-producer-perf-test --producer.config /etc/kafka/consumer.properties --producer-props acks=-1 compression.type=none bootstrap.servers=kafka:9093 --num-records 10 --record-size 20 --throughput 1000 --print-metrics --topic test 
      

      they go through without problems.

      But when the batch is bigger

       

      kafka-producer-perf-test --producer.config /etc/kafka/consumer.properties --producer-props acks=-1 compression.type=none bootstrap.servers=kafka:9093 --num-records 100 --record-size 20 --throughput 1000 --print-metrics --topic test
      

      ... I do not get any messages on the consumer side at all.

      On the other hand, if `kafka-producer-perf-test` throughput is limited like

       

      kafka-producer-perf-test --producer.config /etc/kafka/consumer.properties --producer-props acks=-1 compression.type=none bootstrap.servers=kafka:9093 --num-records 1000 --record-size 10 --throughput 10 --print-metrics --topic test
      

      I can see only the first 20-30 messages in `kafka-console-consumer`. But then it gets stuck (throttled perhaps) and other queued messages never come through.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              kirill.kulikov Kirill Kulikov
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: