[KAFKA-13504] Retry connect internal topics' creation in case of InvalidReplicationFactorException

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.6.0
• Component/s: connect
• Labels: cloudera
• Assignee: Andras Katona
• Reporter: Andras Katona
• Created: 03/Dec/21 14:13
• Updated: 24/May/23 09:11
• Resolved: 24/May/23 09:11

Description

In case the Kafka Broker cluster and the Kafka Connect cluster is started together and Connect would want to create its topics, there's a high chance to fail the creation with InvalidReplicationFactorException.

ERROR org.apache.kafka.connect.runtime.distributed.DistributedHerder [Worker clientId=connect-1, groupId=connect-cluster] Uncaught exception in herder work thread, exiting: 
org.apache.kafka.connect.errors.ConnectException: Error while attempting to create/find topic(s) 'connect-offsets'
...
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 2.

Introducing a retry logic here would make Connect a bit more robust.

The commit uses default.api.timeout.ms and retry.backoff.ms configs to control the retry mechanism.

Issue Links

• links to: GitHub Pull Request #11565