Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-2365 Copycat checklist
  3. KAFKA-2480

Handle non-CopycatExceptions from SinkTasks

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.9.0.0
    • Component/s: KafkaConnect
    • Labels:
      None

      Description

      Currently we catch Throwable in WorkerSinkTask, but we just log the exception. This can lead to data loss because it indicates the messages in the put(records) call probably were not handled properly. We need to decide what the policy for handling these types of exceptions should be – try repeating the same records again, risking duplication? or skip them, risking loss? or kill the task immediately and require intervention since it's unclear what happened?

      SourceTasks don't have the same concern – they can throw other exceptions and as long as we catch them, it is up to the connector to ensure that it does not lose data as a result.

        Attachments

          Activity

            People

            • Assignee:
              ewencp Ewen Cheslack-Postava
              Reporter:
              ewencp Ewen Cheslack-Postava
              Reviewer:
              Gwen Shapira
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: