[KAFKA-13634] Kafka sends false information to the Debezium Connector in case the heartbeat topic does not exists

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Component/s: connect
• Assignee: Unassigned
• Reporter: Andreas Gillmann
• Created: 31/Jan/22 16:25
• Updated: 16/Mar/22 01:38

Description

I had already submitted this issue to RedHat (https://issues.redhat.com/browse/DBZ-4521). However, RedHat replied that they only send the message and the topic name to Kafka and Kafka returns an incorrect response.

 

Heartbeat fails because the heartbeat topic does not exist and the auto.create.topics parameter is set to false, but the status of the connector remains 'Running'.

 

Our expectations:

The connector should go to status 'failed' after a few failed heartbeat attempts. At the moment - status = 'running' - everything looks good and we wonder why no data arrives in Kafka.

 

Steps to reproduce:

Prerequisites:

• Debezium connector with heartbeat configuration ( "heartbeat.topics.prefix" is set)
• Heartbeat topic on Kafka side does NOT exists
• Parameter auto.create.topics is disabled (=false)

Execution:

• Connector was created
• Connector starts the heartbeat
• Connector logs the following entry: [WARN] 2022-01-06 13:25:30,094 [kafka-producer-network-thread | connector-producer-itgermany.pmops.dispoapp.gateway.DE-350.debeziumconnector-0] org.apache.kafka.clients.NetworkClient handleSuccessfulResponse - [Producer clientId=connector-producer-itgermany.pmops.dispoapp.gateway.DE-350.debeziumconnector-0] Error while fetching metadata with correlation id 38409 : {itgermany.pmops.dispoapp.gateway.heartbeat.GatewayDepot35=UNKNOWN_TOPIC_OR_PARTITION}
• Connector status remains 'RUNNING'
• NO Data comes to Kafka via the Connector

Troubleshooting:

• Re-creating the connector caused data to be sent to Kafka for a short time (until the heartbeat was repeated).

Issue Links

• is caused by: KAFKA-12990 Infinite Loop in producing event to kafka