[KAFKA-4385] producer is sending too many unnecessary meta data request if the meta data for a topic is not available and "auto.create.topics.enable" =false

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.9.0.1
• Component/s: producer
• Assignee: Unassigned
• Reporter: Jun Yao
• Created: 06/Nov/16 04:34
• Updated: 15/Jan/23 23:12

Description

All current kafka-client producer implementation (<= 0.10.1.0),
When sending a msg to a topic, it will first check if meta data for this topic is available or not,
when not available, it will set "metadata.requestUpdate()" and wait for meta data from brokers,

The thing is inside "org.apache.kafka.clients.Metadata.awaitUpdate()", it's already doing a "while (this.version <= lastVersion)" loop waiting for new version response,
So the loop inside "org.apache.kafka.clients.producer.KafkaProducer.waitOnMetadata() is not needed,

When "auto.create.topics.enable" is false, sending msgs to a non-exist topic will trigger too many meta requests, everytime a metadata response is returned, because it does not contain the metadata for the topic, it's going to try again until TimeoutException is thrown;
This is a waste and sometimes causes too much overhead when unexpected msgs are arrived.

Issue Links

• is related to: KAFKA-2948 Kafka producer does not cope well with topic deletions
• links to: GitHub Pull Request #2107
• links to: GitHub Pull Request #2107