Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-12270

Kafka Connect may fail a task when racing to create topic

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.6.0, 2.7.0, 2.8.0
    • Fix Version/s: 2.8.0, 2.7.1, 2.6.2
    • Component/s: KafkaConnect
    • Labels:
      None

      Description

      When a source connector configured with many tasks and to use the new topic creation feature is run, it is possible that multiple tasks will attempt to write to the same topic, will see that the topic does not exist, and then race to create the topic. The topic is only created once, but some tasks might fail with:

      org.apache.kafka.connect.errors.ConnectException: Task failed to create new topic (name=TOPICX, numPartitions=8, replicationFactor=3, replicasAssignments=null, configs={cleanup.policy=delete}). Ensure that the task is authorized to create topics or that the topic exists and restart the task
        at org.apache.kafka.connect.runtime.WorkerSourceTask.maybeCreateTopic(WorkerSourceTask.java:436)
        at org.apache.kafka.connect.runtime.WorkerSourceTask.sendRecords(WorkerSourceTask.java:364)
        at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:264)
        at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:185)
        at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:235)
      ... 

      The reason appears to be that the WorkerSourceTask throws an exception if the topic creation failed, and does not account for the fact that the topic may have been created between the time the WorkerSourceTask lists existing topics and tries to create the topic.

       

      See in particular: https://github.com/apache/kafka/blob/5c562efb2d76407011ea88c1ca1b2355079935bc/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSourceTask.java#L415-L423

       

      This is only an issue when using topic creation settings in the source connector configuration, and when running multiple tasks that write to the same topic.

      The workaround is to create the topics manually before starting the connector, or to simply restart the failed tasks using the REST API.

        Attachments

          Activity

            People

            • Assignee:
              rhauch Randall Hauch
              Reporter:
              rhauch Randall Hauch
              Reviewer:
              Konstantine Karantasis
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: