[KAFKA-6626] Performance bottleneck in Kafka Connect sendRecords

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Abandoned
• Affects Version/s: 1.0.0
• Component/s: connect
• Assignee: Unassigned
• Reporter: Maciej Bryński
• Created: 08/Mar/18 07:35
• Updated: 03/Oct/18 13:04
• Resolved: 03/Oct/18 13:04

Description

Kafka Connect is using IdentityHashMap for storing records.

https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSourceTask.java#L239

Unfortunately this solution is very slow (2-4 times slower than normal HashMap / HashSet).

Benchmark result (code in attachment). 

Identity 4220
Set 2115
Map 1941
Fast Set 2121

Things are even worse when using default GC configuration
(-server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35  -Djava.awt.headless=true)

Identity 7885
Set 2364
Map 1548
Fast Set 1520

Java version

java version "1.8.0_152"
Java(TM) SE Runtime Environment (build 1.8.0_152-b16)
Java HotSpot(TM) 64-Bit Server VM (build 25.152-b16, mixed mode)

This problem is greatly slowing Kafka Connect.