[KAFKA-8623] KafkaProducer possible deadlock when sending to different topics

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.2.1
• Fix Version/s: 2.3.0
• Component/s: producer
• Assignee: Kun Song
• Reporter: Alexander Bagiev
• Created: 02/Jul/19 10:50
• Updated: 12/Jun/20 02:49
• Resolved: 10/Feb/20 17:22

Description

Project with bug reproduction: https://github.com/abagiev/kafka-producer-bug

It was found that sending two messages in two different topics in a row results in hanging of KafkaProducer for 60s and the following exception:

org.springframework.kafka.core.KafkaProducerException: Failed to send; nested exception is org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
	at org.springframework.kafka.core.KafkaTemplate.lambda$buildCallback$0(KafkaTemplate.java:405) ~[spring-kafka-2.2.7.RELEASE.jar:2.2.7.RELEASE]
	at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:877) ~[kafka-clients-2.0.1.jar:na]
	at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:803) ~[kafka-clients-2.0.1.jar:na]
	at org.springframework.kafka.core.DefaultKafkaProducerFactory$CloseSafeProducer.send(DefaultKafkaProducerFactory.java:444) ~[spring-kafka-2.2.7.RELEASE.jar:2.2.7.RELEASE]
	at org.springframework.kafka.core.KafkaTemplate.doSend(KafkaTemplate.java:381) ~[spring-kafka-2.2.7.RELEASE.jar:2.2.7.RELEASE]
	at org.springframework.kafka.core.KafkaTemplate.send(KafkaTemplate.java:193) ~[spring-kafka-2.2.7.RELEASE.jar:2.2.7.RELEASE]
...

It looks like KafkaProducer requests two times for meta information for each topic and hangs just before second request due to some deadlock. When 60s pass TimeoutException is thrown and meta information is requested/received immediately (but after exception has been already thrown).

The issue in the example project is reproduced every time; and the use case is trivial.
This is a critical bug for us.

Issue Links

• is related to: KAFKA-7831 Consumer SubscriptionState missing synchronization
• relates to: KAFKA-10154 Issue in updating metadata if not exists during sending message to different topics