[KAFKA-6705] producer.send() should not block due to metadata not available

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Assignee: Dong Lin
• Reporter: Dong Lin
• Created: 23/Mar/18 04:35
• Updated: 18/Oct/21 08:54
• Resolved: 24/Apr/18 21:26

Description

Currently producer.send() may block on metadata for up to max.block.ms. This behavior is well documented but it is a bit sub-optimal. Since we return a future we should be able to make producer.send() completely non-blocking. One idea is to simply insert the record into a global queue shared across all partitions, and let the sender thread fetch record from this queue and send to broker.

Issue Links

• causes: LOG4J2-3173 KafkaAppender for log4j2 prevents an app from terminating
• is related to: KAFKA-3539 KafkaProducer.send() may block even though it returns the Future
• is related to: KAFKA-6127 Streams should never block infinitely