[KAFKA-10901] Lock contention on high produce rate causing cluster degregation

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.5.0
• Component/s: producer
• Environment: broker: version 2.5.0 with 8 cores,32gb, hdd ; producer: Sarama producer version 1.5.2(go) with 500ms linger and 2mb batch-size
• Assignee: Unassigned
• Reporter: ilya morgenshtern
• Created: 05/Jan/21 10:41
• Updated: 06/Jan/21 18:18

Description

scaling up (20 -> 40) producers causing idle percentage to drop from 70-80% into 0-1 %, the request queue size to increase by 200%, and overall producers latency increased by 700%.
also, the CPU usage dropped by 30%

after we ran some profiling we saw that there is high lock contention on the write requests but, CPU remained low, we didn't we any strange activity in the disk write/read/IOPS only the other way around because everything became slower the cluster processed much fewer data.

in comparison when there were 20 producers, you can see that the ratio of produce/fetch is

 

from limited observation, we saw the number of the produce request from this upscaled producer increased from 1500 to 2500(150 per broker) per sec, but overall produce request in the cluster remained the same, on the other hand, the number of fetch requests decreased by 50%

to fix the issue we increased this specific producer linger.ms from 500ms to 1000ms and suddenly the whole cluster became healthy.