[KAFKA-7422] Huge memory consumption (leak?) by Kafka Producer

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.10.1.0
• Component/s: producer
• Environment: OS - AIX 7.1, Java - IBM Java 1.8
• Assignee: Unassigned
• Reporter: Shantanu Deshmukh
• Created: 19/Sep/18 07:55
• Updated: 15/Oct/18 09:32

Description

Hello,
 
We have a 3 broker Kafka 0.10.1.0 deployment in production. There are some applications which have Kafka Producers embedded in them which send application logs to a topic. This topic has 10 partitions with replication factor of 3.
 
We are observing that memory usage on some of these application servers keep shooting through the roof intermittently. After taking heapdump we found out that top suspects were:
---------------------

org.apache.kafka.common.network.Selector -

occupies 352,519,104 (24.96%) bytes. The memory is accumulated in one instance of "byte[]" loaded by "<system class loader>".
org.apache.kafka.common.network.KafkaChannel -
occupies 352,527,424 (24.96%) bytes. The memory is accumulated in one instance of "byte[]" loaded by "<system class loader>"

---------------------

Both of these were holding about 352MB of space. 3 such instances, so they were consuming about 1.2GB of memory.

Now regarding usage of producers. Not a huge amount of logs are being sent to Kafka cluster. It is about 200 msgs/sec. Only one producer object is being used throughout application. Async send function is used.

What could be the cause of such huge memory usage? Is this some sort of memory leak in this specific Kafka version?

Here's producer config being used at the present.

>>>>>>>>>>>>>>>>
kafka.bootstrap.servers=x.x.x.x:9092,x.x.x.x:9092,x.x.x.x:9092
{{ kafka.acks=0}}
{{ kafka.key.serializer=org.apache.kafka.common.serialization.StringSerializer}}
{{ kafka.value.serializer=org.apache.kafka.common.serialization.StringSerializer}}
kafka.max.block.ms=1000
kafka.request.timeout.ms=1000
{{ kafka.max.in.flight.requests.per.connection=1}}
{{ kafka.retries=0}}
{{ kafka.compression.type=gzip}}
{{ kafka.security.protocol=SSL}}
{{ kafka.ssl.truststore.location=/data/kafka/kafka-server-truststore.jks}}
{{ kafka.ssl.truststore.password=XXXXXX}}
{{ logger.level=INFO }}<<<<<<<<<<<<<<
Attaching heapdump for your perusal.