Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-2365 Copycat checklist
  3. KAFKA-2480

Handle non-CopycatExceptions from SinkTasks

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.9.0.0
    • connect
    • None

    Description

      Currently we catch Throwable in WorkerSinkTask, but we just log the exception. This can lead to data loss because it indicates the messages in the put(records) call probably were not handled properly. We need to decide what the policy for handling these types of exceptions should be – try repeating the same records again, risking duplication? or skip them, risking loss? or kill the task immediately and require intervention since it's unclear what happened?

      SourceTasks don't have the same concern – they can throw other exceptions and as long as we catch them, it is up to the connector to ensure that it does not lose data as a result.

      Attachments

        Activity

          People

            ewencp Ewen Cheslack-Postava
            ewencp Ewen Cheslack-Postava
            Gwen Shapira Gwen Shapira
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: