[KAFKA-12990] Infinite Loop in producing event to kafka

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Environment: Kafka 2.4.1 ; Kafka Connect 5.4.3
• Assignee: Unassigned
• Reporter: Piotr Rybarczyk
• Created: 24/Jun/21 12:34
• Updated: 16/Mar/22 01:43

Description

Marked as a bug, but depends on what configuration means.

We are using AWS RDS as a Postgres database based on Postgres 11 version.
We are using Debezium 1.4.1 version with pgoutput and outbox router. 
Recently, I've been testing what will happen if the topic will not be configured in Kafka.
Normally, in that case, connector login warns & stuck in an infinite loop as the task is unable to produce an event to Kafka:

 WARN [Producer clientId=connector-producer-piotrek-test-3.outbox_events_debezium_connector-0] Error while fetching metadata with correlation id 3884 : {piotrek-test-123.partners.feature-flag-events=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)

I've confirmed with the Debezium team that this issue is not in the connector logic scope. From my guess, it needs to be on Kafka or Kafka Connect level.  

The worst thing is that even when I've disabled the connector, & task logs still have been produced to console no matter what I've done. So in that case, we can not really on task statuses to check is pipeline working, we need to verify it on logs level. 

More info in

• Debezium Issue
  - Slack

Issue Links

• causes: KAFKA-13634 Kafka sends false information to the Debezium Connector in case the heartbeat topic does not exists
• relates to: KAFKA-10340 Source connectors should report error when trying to produce records to non-existent topics instead of hanging forever