[KAFKA-15041] Source Connector auto topic creation fails when topic is deleted and brokers don't support auto topic creation

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: connect
• Assignee: Sagar Rao
• Reporter: Sagar Rao
• Created: 31/May/23 10:34
• Updated: 31/May/23 10:46

Description

KIP-158  allows the source connectors to create topics even when the broker doesn't allow to do so. It does so by checking for every record if a topic needs to be created https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractWorkerSourceTask.java#L500. To not always keep checking for topic presence via admin topics, it also maintains a cache of the topics that it has created and doesn't create those anymore. This helps to create topics when brokers don't support automatic topic creation.

However, lets say the topic gets created initially and later on gets deleted while the connector is still running and the brokers don't support automatic topic creation. For such cases, the connector has cached the topic it has already created and wouldn't recreate it because the cache never updates and since the broker doesn't support topic creation, the logs would just be full of messages like 

 

Error while fetching metadata with correlation id 3260 : {connect-test=UNKNOWN_TOPIC_OR_PARTITION}

 

This can become a problem on environments where brokers don't allow topic creation. We need a way to refresh the topics cache for such cases.