[KAFKA-9063] KAfka producer throughput drops with number of topics even when producer is producing on one topic

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.2.1
• Assignee: Unassigned
• Reporter: Pradeep Bansal
• Created: 18/Oct/19 04:54
• Updated: 23/Oct/19 18:26

Description

5 broker cluster

Topic partitions  =  1

Replication factor = 3

Ac mode = all

Send type = Asynchronous

Message size  =  100 bytes

Log compaction  =  Enabled

1. We started throughput test with 1 topic and number of topics present in cluster at that time were 1 (excluding already existing topics 1000)
2. We left it to run for about 2 hours (The mean throughput we observed during this period was 54k msgs/sec)
3. After this, we started creating 10,000 topics one by one using a script
4. We noted throughput values after creating 100 topics and after 200,300,400…so on till 10,000 were created
5. After all 10,000 topics were created we left test to run for another 1 hr.

 During the entire duration, we were producing only on a single topic.

 

What could be causing producer throughput drop by just adding number of topics?