Major Description
KafkaLog4jAppender instantiates a KafkaProducer to append log entries to a Kafka topic. The producer.send operation may need to acquire locks during its execution. This can result in deadlocks when a log entry from the producer network thread is also at a log level that results in the entry being appended to a Kafka topic (KAFKA-6415).
https://github.com/apache/kafka/pull/11691 enables idempotence by default, and it introduced another place where the producer network thread can hit a deadlock. When calling TransactionManger#maybeAddPartition, the producer network thread will wait on the TransactionManager lock, and a deadlock can happen if TransactionManager also logs at INFO level. This is causing system test failures in log4j_appender_test.py
Similar to KAFKA-6415, a workaround will be setting log level to WARN for TransactionManager in VerifiableLog4jAppender.
Attachments
Issue Links
- links to