[KAFKA-2202] ConsumerPerformance reports a throughput much higher than the actual one

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.8.2.0
• Fix Version/s: 0.9.0.0
• Component/s: tools
• Assignee: Manikumar
• Reporter: Micael Capitão
• Created: 19/May/15 15:01
• Updated: 11/Aug/15 03:57
• Resolved: 11/Aug/15 03:57

Description

I've been using the kafka.tools.ConsumerPerformance tool for some benchmarking until in one of my tests I got a throughput much higher than the supported by my network interface.
The test consisted in consuming around ~4900 MB from one topic using one consumer with one thread. The reported throughput reported was ~1400 MB/s which surpasses the 10 Gbps of the network. The time for the whole operation was ~8 seconds, which should correspond to a throughput of ~612 MB/s.
Digging the ConsumerPerformance code, I've found this at line 73:

val elapsedSecs = (endMs - startMs - config.consumerConfig.consumerTimeoutMs) / 1000.0

The consumerTimeoutMs defined as 5000 at line 131 is always considered leading to wrong results.

This bug seems to be related to this one https://issues.apache.org/jira/browse/KAFKA-1828

Issue Links

• duplicates: KAFKA-1828 [ConsumerPerformance] the test result is negative number