Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
Description
Currently we catch Throwable in WorkerSinkTask, but we just log the exception. This can lead to data loss because it indicates the messages in the put(records) call probably were not handled properly. We need to decide what the policy for handling these types of exceptions should be – try repeating the same records again, risking duplication? or skip them, risking loss? or kill the task immediately and require intervention since it's unclear what happened?
SourceTasks don't have the same concern – they can throw other exceptions and as long as we catch them, it is up to the connector to ensure that it does not lose data as a result.