Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-8586

Source task producers silently fail to send records

    XMLWordPrintableJSON

    Details

      Description

      The Connect framework marks source records as successfully sent when they are dispatched to the producer, instead of when they are actually sent to Kafka. This is assumed to be good enough since the Connect framework sets up its producer to use infinite retries on retriable errors, but in the case of an authorization or authentication failure with a secured Kafka broker, the errors aren't retriable and cause the producer to invoke its send callback with an exception and then give up on sending the message. This is a problem since the callback currently used by the WorkerSourceTask class when it invokes Producer.send(...) logs the exception and does nothing else. This leads to data loss since the source offsets for those failed records are committed, and the status of the task is never affected so users may not even know that something is wrong unless they check the worker log files or notice that data isn't flowing into Kafka. Until and unless someone does notice that something's wrong, the task will continue processing records and committing offsets, even though nothing is making it into Kafka.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ChrisEgerton Chris Egerton
                Reporter:
                ChrisEgerton Chris Egerton
                Reviewer:
                Randall Hauch
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: