Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-10340

Source connectors should report error when trying to produce records to non-existent topics instead of hanging forever

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.5.1, 2.7.0, 2.6.1, 2.8.0
    • 2.7.1, 2.6.2, 2.8.1, 3.0.0
    • KafkaConnect
    • None

    Description

      Currently, a source connector will blindly attempt to write a record to a Kafka topic. When the topic does not exist, its creation is controlled by the auto.create.topics.enable config on the brokers. When auto.create is disabled, the producer.send() call on the Connect worker will hang indefinitely (due to the "infinite retries" configuration for said producer). In setups where this config is usually disabled, the source connector simply appears to hang and not produce any output.

      It is desirable to either log an info or an error message (or inform the user somehow) that the connector is simply stuck waiting for the destination topic to be created. When the worker has permissions to inspect the broker settings, it can use the listTopics and describeConfigs API in AdminClient to check if the topic exists, the broker can auto.create.topics.enable topics, and if these cases do not exist, either throw an error.

      With the recently merged KIP-158, this becomes even more specific a corner case: when topic creation settings are enabled, the worker should handle the corner case where topic creation is disabled, auto.create.topics.enable is set to false and topic does not exist.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            ChrisEgerton Chris Egerton
            wicknicks Arjun Satish
            Randall Hauch Randall Hauch
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment