Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-9982

[kafka-connect] Source connector does not guarantee at least once delivery

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Not A Bug
    • Affects Version/s: 2.5.0
    • Fix Version/s: None
    • Component/s: KafkaConnect
    • Labels:
      None

      Description

      In kafka-connect runtime, the WorkerSourceTask is responsible for sending records to the destination topics and managing the source offset commit. Committed offsets are then used later for recovery of tasks during rebalance or restart.

      But there are two concerns when looking into the WorkerSourceTask implementation:

      • When producer fail to send records, there's no retry but just skipping offset commit and then execute next loop (poll for new records)
      • The offset commit and effectively sending records over network are in fact asynchronous, which means the offset commit could happen before records are received by brokers, and a rebalance/restart in this gap could lead to message loss.

      The conclusion is thus that the source connector does not support at least once semantics by default (without the plugin implementation making extra effort itself). I consider this as a bug.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              q.xu Qinghui Xu
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: