[KAFKA-2517] Performance Regression post SSL implementation

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.9.0.0
• Assignee: Ismael Juma
• Reporter: Ben Stopford
• Reviewer: Jun Rao
• Created: 04/Sep/15 11:46
• Updated: 02/Oct/15 21:46
• Resolved: 02/Oct/15 21:46

Description

It would appear that we incurred a performance regression on submission of the SSL work affecting the performance of the new Kafka Consumer.

Running with 1KB messages. Macbook 2.3 GHz Intel Core i7, 8GB, APPLE SSD SM256E. Single server instance. All local.

kafka-consumer-perf-test.sh ... --messages 3000000 --new-consumer

Pre-SSL changes (commit 503bd36647695e8cc91893ffb80346dd03eb0bc5)
Steady state throughputs = 234.8 MB/s
(2861.5913, 234.8261, 3000596, 246233.0543)

Post-SSL changes (commit 13c432f7952de27e9bf8cb4adb33a91ae3a4b738)
Steady state throughput = 178.1 MB/s
(2861.5913, 178.1480, 3000596, 186801.7182)

Implication is a 25% reduction in consumer throughput for these test conditions.

This appears to be caused by the use of PlaintextTransportLayer rather than SocketChannel in FileMessageSet.writeTo() meaning a zero copy transfer is not invoked.

Switching to the use of a SocketChannel directly in FileMessageSet.writeTo() yields the following result:
Steady state throughput = 281.8 MB/s
(2861.5913, 281.8191, 3000596, 295508.7650)

Issue Links

• relates to: KAFKA-2431 Test SSL/TLS impact on performance