Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-9063

KAfka producer throughput drops with number of topics even when producer is producing on one topic

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.2.1
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      5 broker cluster

      Topic partitions  =  1

      Replication factor = 3

      Ac mode = all

      Send type = Asynchronous

      Message size  =  100 bytes

      Log compaction  =  Enabled

      1. We started throughput test with 1 topic and number of topics present in cluster at that time were 1 (excluding already existing topics 1000)
      2. We left it to run for about 2 hours (The mean throughput we observed during this period was 54k msgs/sec)
      3. After this, we started creating 10,000 topics one by one using a script
      4. We noted throughput values after creating 100 topics and after 200,300,400…so on till 10,000 were created
      5. After all 10,000 topics were created we left test to run for another 1 hr.

       During the entire duration, we were producing only on a single topic.

       

      What could be causing producer throughput drop by just adding number of topics?

        Attachments

        1. image-2019-10-18-10-22-40-372.png
          42 kB
          Pradeep Bansal

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              bansalp Pradeep Bansal
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: