Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-6738

Kafka Connect handling of bad data

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 1.1.0
    • 2.0.0
    • KafkaConnect
    • None

    Description

      Kafka Connect connectors and tasks fail when they run into an unexpected situation or error, but the framework should provide more general "bad data handling" options, including (perhaps among others):

      1. fail fast, which is what we do today (assuming connector actually fails and doesn't eat errors)
      2. retry (possibly with configs to limit)
      3. drop data and move on
      4. dead letter queue

      This needs to be addressed in a way that handles errors from:

      1. The connector itself (e.g. connectivity issues to the other system)
      2. Converters/serializers (bad data, unexpected format, etc)
      3. SMTs
      4. Ideally the framework as well, though we obviously want to fix known bugs anyway

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            wicknicks Arjun Satish
            rhauch Randall Hauch
            Ewen Cheslack-Postava Ewen Cheslack-Postava
            Votes:
            1 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment