Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-4385

producer is sending too many unnecessary meta data request if the meta data for a topic is not available and "auto.create.topics.enable" =false

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.9.0.1
    • Fix Version/s: None
    • Component/s: producer
    • Labels:
      None

      Description

      All current kafka-client producer implementation (<= 0.10.1.0),
      When sending a msg to a topic, it will first check if meta data for this topic is available or not,
      when not available, it will set "metadata.requestUpdate()" and wait for meta data from brokers,

      The thing is inside "org.apache.kafka.clients.Metadata.awaitUpdate()", it's already doing a "while (this.version <= lastVersion)" loop waiting for new version response,
      So the loop inside "org.apache.kafka.clients.producer.KafkaProducer.waitOnMetadata() is not needed,

      When "auto.create.topics.enable" is false, sending msgs to a non-exist topic will trigger too many meta requests, everytime a metadata response is returned, because it does not contain the metadata for the topic, it's going to try again until TimeoutException is thrown;
      This is a waste and sometimes causes too much overhead when unexpected msgs are arrived.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                Jun Yao Jun Yao
              • Votes:
                2 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated: