Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-10356

Handle accidental deletion of sink-topics as exceptional failure

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: streams
    • Labels:
      None

      Description

      Today when sink topics are deleted, the producer's send callback would eventually return the UnknownTopicOrPartition exception after configured max.delivery.ms whose default is 2min if EOS is not enabled (otherwise its Integer.MAX_VALUE). Then in Streams implementation the exception would be handled by ProductionExceptionHandler which by default would treat it as `FAIL` and would cause the thread to die. If it treat it is CONTINUE then it would be silently ignored and the sent records are lost silently.

      We should improve this situation in Streams by special-handling UnknownTopicOrPartition exception and trigger a rebalance as well, and then in leader we can also check if the sink topic metadata exists, just like source topic, and then follow the same logic as in https://issues.apache.org/jira/browse/KAFKA-10355.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                cadonna Bruno Cadonna
                Reporter:
                guozhang Guozhang Wang
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: