Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-6250

Kafka Connect requires permission to create internal topics even if they exist

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.11.0.1, 1.0.0
    • Fix Version/s: 1.1.0
    • Component/s: KafkaConnect
    • Labels:
      None

      Description

      When using Kafka Connect with a cluster that doesn't allow the user to create topics (due to ACL configuration), Connect fails when trying to create its internal topics, even if these topics already exist.

      This happens specifically when using hosted Aiven Kafka, which does not permit creation of topics via the Kafka Admin Client API.

      The problem is that Connect tries to create the topics, and ignores some specific errors such as topics that already exist, but not authorization errors.

      This is what happens:

      2017-11-21 15:57:24,176 [DistributedHerder] ERROR DistributedHerder:206 - Uncaught exception in herder work thread, exiting:
      org.apache.kafka.connect.errors.ConnectException: Error while attempting to create/find topic(s) 'connect-offsets'
      	at org.apache.kafka.connect.util.TopicAdmin.createTopics(TopicAdmin.java:245)
      	at org.apache.kafka.connect.storage.KafkaOffsetBackingStore$1.run(KafkaOffsetBackingStore.java:99)
      	at org.apache.kafka.connect.util.KafkaBasedLog.start(KafkaBasedLog.java:126)
      	at org.apache.kafka.connect.storage.KafkaOffsetBackingStore.start(KafkaOffsetBackingStore.java:109)
      	at org.apache.kafka.connect.runtime.Worker.start(Worker.java:146)
      	at org.apache.kafka.connect.runtime.AbstractHerder.startServices(AbstractHerder.java:99)
      	at org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:194)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.ClusterAuthorizationException: Cluster authorization failed.
      	at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
      	at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
      	at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
      	at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:213)
      	at org.apache.kafka.connect.util.TopicAdmin.createTopics(TopicAdmin.java:226)
      	... 11 more
      Caused by: org.apache.kafka.common.errors.ClusterAuthorizationException: Cluster authorization failed.
      

        Attachments

          Activity

            People

            • Assignee:
              gavriep Gavrie Philipson
              Reporter:
              gavriep Gavrie Philipson
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: