Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-10606

Auto create non-existent topics when fetching metadata for all topics

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.7.0
    • Component/s: None
    • Labels:
      None

      Description

      The "allow auto topic creation" flag is hardcoded to be true for the fetch-all-topic metadata request:

      https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/requests/MetadataRequest.java#L37

      In the below code, annotation claims that "This never causes auto-creation". It it NOT true and auto topic creation still gets triggered under some circumstances. So, this is a bug that needs to be fixed.

      https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/requests/MetadataRequest.java#L68

      For example, the bug could be manifested in the below situation:

      A topic T is being deleted and a request to fetch metadata for all topics gets sent to one broker. The broker reads names of all topics from its metadata cache (shown below).

      https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/KafkaApis.scala#L1196

      Then the broker authorizes all topics and makes sure that they are allowed to be described. Then the broker tries to get metadata for every authorized topic by reading the metadata cache again, once for every topic (show below).

      https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/KafkaApis.scala#L1240

      However, the metadata cache could have been updated while the broker was authorizing all topics and topic T and its metadata no longer exist in the cache since the topic got deleted and metadata update requests eventually got propagated from the controller to all brokers. So, at this point, when the broker tries to get metadata for topic T from its cache, it realizes that it does not exist and the broker tries to "auto create" topic T since the allow-auto-topic-creation flag was set to true in all the fetch-all-topic metadata requests.

      I think this bug exists since "metadataRequest.allowAutoTopicCreation" was introduced.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                andrewlincong@gmail.com Lincong Li
                Reporter:
                andrewlincong@gmail.com Lincong Li
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: