Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-2948

Kafka producer does not cope well with topic deletions

Agile BoardAttach filesAttach ScreenshotVotersStop watchingWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.9.0.0
    • 0.10.1.0
    • producer
    • None

    Description

      Kafka producer gets metadata for topics when send is invoked and thereafter it attempts to keep the metadata up-to-date without any explicit requests from the client. This works well in static environments, but when topics are added or deleted, list of topics in Metadata grows but never shrinks. Apart from being a memory leak, this results in constant requests for metadata for deleted topics.

      We are running into this issue with the Confluent REST server where topic deletion from tests are filling up logs with warnings about unknown topics. Auto-create is turned off in our Kafka cluster.

      I am happy to provide a fix, but am not sure what the right fix is. Does it make sense to remove topics from the metadata list when UNKNOWN_TOPIC_OR_PARTITION response is received if there are no outstanding sends? It doesn't look very straightforward to do this, so any alternative suggestions are welcome.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            rsivaram Rajini Sivaram
            rsivaram Rajini Sivaram
            Ewen Cheslack-Postava Ewen Cheslack-Postava
            Votes:
            2 Vote for this issue
            Watchers:
            8 Stop watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment