[KAFKA-16508] Infinte loop if output topic does not exisit

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: streams
• Assignee: Unassigned
• Reporter: Matthias J. Sax
• Created: 10/Apr/24 15:50
• Updated: 10/Apr/24 15:50

Description

Kafka Streams supports `ProductionExceptionHandler` to drop records on error when writing into an output topic.

However, if the output topic does not exist, the corresponding error cannot be skipped over because the handler is not called.

The issue is, that the producer internally retires to fetch the output topic metadata until it times out, an a `TimeoutException` (which is a `RetriableException`) is returned via the registered `Callback`. However, for `RetriableException` there is different code path and the `ProductionExceptionHandler` is not called.

In general, Kafka Streams correctly tries to handle as many errors a possible internally, and a `RetriableError` falls into this category (and thus there is no need to call the handler). However, for this particular case, just retrying does not solve the issue – it's unclear if throwing a retryable `TimeoutException` is actually the right thing to do for the Producer? Also not sure what the right way to address this ticket would be (currently, we cannot really detect this case, except if we would do some nasty error message String comparison what sounds hacky...)