Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
We have observed, using two entirely different tools, that a simple Kafka application (read 1 topic and immediately produce to another) suffers substantial throughput degradation when scaling up topics. Below is the output of one of these tools, showing that going from 1 partition to 1000 partitions yields a ~30% throughput decrease when messages are 100 bytes long.
Using the same two tools, we observed that increasing the message size to 512 bytes yields a throughput increase of ~20% going from 1 topic partition to 1000 topic partitions with all other variables held constant.
| Kafka Core Testing | ||||||||||
| Enable Transaction | Batch Size (b) | Linger (ms) | Max Inflight | Commit Interval (ms) | Num Records | Record Size (b) | Num Input Topics | 1 Partition MB/s | 1000 Partitions MB/s | MB/s delta |
| FALSE | 16384 | 100 | 5 | 1000 | 20000000 | 100 | 1 | 45.633625 | 31.482193 | -31.01% |
| FALSE | 16384 | 100 | 5 | 1000 | 20000000 | 512 | 1 | 70.217902 | 85.319107 | 21.51% |