[KAFKA-10356] Handle accidental deletion of sink-topics as exceptional failure

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Component/s: streams
• Assignee: Bruno Cadonna
• Reporter: Guozhang Wang
• Created: 04/Aug/20 18:51
• Updated: 12/Jul/24 00:58
• Resolved: 12/Jul/24 00:58

Description

Today when sink topics are deleted, the producer's send callback would eventually return the UnknownTopicOrPartition exception after configured max.delivery.ms whose default is 2min if EOS is not enabled (otherwise its Integer.MAX_VALUE). Then in Streams implementation the exception would be handled by ProductionExceptionHandler which by default would treat it as `FAIL` and would cause the thread to die. If it treat it is CONTINUE then it would be silently ignored and the sent records are lost silently.

We should improve this situation in Streams by special-handling UnknownTopicOrPartition exception and trigger a rebalance as well, and then in leader we can also check if the sink topic metadata exists, just like source topic, and then follow the same logic as in https://issues.apache.org/jira/browse/KAFKA-10355.

Issue Links

• is related to: KAFKA-10355 Handle accidental deletion of source-topics as exceptional failure