[KAFKA-4557] ConcurrentModificationException in KafkaProducer event loop

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.10.1.0
• Fix Version/s: 0.10.2.0
• Component/s: clients
• Labels: reliability
• Assignee: Rajini Sivaram
• Reporter: Sergey Alaev
• Created: 19/Dec/16 15:07
• Updated: 27/Jan/17 23:28
• Resolved: 27/Jan/17 23:28

Description

Under heavy load, Kafka producer can stop publishing events. Logs below.

[2016-12-19T15:01:28.779Z] [sgs] [kafka-producer-network-thread | producer-3] [NetworkClient] [] [<none>] [] [DEBUG]: Disconnecting from node 2 due to request timeout.
[2016-12-19T15:01:28.793Z] [sgs] [kafka-producer-network-thread | producer-3] [KafkaProducerClient] [] [<none>] [1B2M2Y8Asg] [WARN]: Error sending message to Kafka
org.apache.kafka.common.errors.NetworkException: The server disconnected before a response was received.
[2016-12-19T15:01:28.838Z] [sgs] [kafka-producer-network-thread | producer-3] [KafkaProducerClient] [] [<none>] [1B2M2Y8Asg] [WARN]: Error sending message to Kafka
org.apache.kafka.common.errors.NetworkException: The server disconnected before a response was received. (#2 from 2016-12-19T15:01:28.793Z)

--------------------------------

[2016-12-19T15:01:28.956Z] [sgs] [kafka-producer-network-thread | producer-3] [KafkaProducerClient] [] [<none>] [1B2M2Y8Asg] [WARN]: Error sending message to Kafka
org.apache.kafka.common.errors.TimeoutException: Expiring 46 record(s) for events-deadletter-0 due to 30032 ms has passed since batch creation plus linger time (#285 from 2016-12-19
T15:01:28.793Z)
[2016-12-19T15:01:28.956Z] [sgs] [kafka-producer-network-thread | producer-3] [SgsService] [] [<none>] [1B2M2Y8Asg] [WARN]: Error writing signal to Kafka deadletter queue
org.apache.kafka.common.errors.TimeoutException: Expiring 46 record(s) for events-deadletter-0 due to 30032 ms has passed since batch creation plus linger time (#286 from 2016-12-19
T15:01:28.793Z)
[2016-12-19T15:01:28.960Z] [sgs] [kafka-producer-network-thread | producer-3] [Sender] [] [<none>] [1B2M2Y8Asg] [ERROR]: Uncaught error in kafka producer I/O thread:
java.util.ConcurrentModificationException: null
at java.util.ArrayDeque$DeqIterator.next(ArrayDeque.java:643) ~[na:1.8.0_45]
at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortExpiredBatches(RecordAccumulator.java:242) ~[kafka-clients-0.10.1.0.jar:na]
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:212) ~[kafka-clients-0.10.1.0.jar:na]
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:135) ~[kafka-clients-0.10.1.0.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
[2016-12-19T15:01:28.981Z] [sgs] [kafka-producer-network-thread | producer-3] [NetworkClient] [] [<none>] [1B2M2Y8Asg] [WARN]: Error while fetching metadata with correlation id 28711 :

{events-deadletter=LEADER_NOT_AVAILABLE}

Issue Links

• links to: GitHub Pull Request #2449