Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-10000

Atomic commit of source connector records and offsets

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: KafkaConnect
    • Labels:

      Description

      It'd be nice to be able to configure source connectors such that their offsets are committed if and only if all records up to that point have been ack'd by the producer. This would go a long way towards EOS for source connectors.

       

      This differs from https://issues.apache.org/jira/browse/KAFKA-6079, which is marked as WONTFIX since it only concerns enabling the idempotent producer for source connectors and is not concerned with source connector offsets.

      This also differs from https://issues.apache.org/jira/browse/KAFKA-6080, which had a lot of discussion around allowing connector-defined transaction boundaries. The suggestion in this ticket is to only use source connector offset commits as the transaction boundaries for connectors; allowing connector-specified transaction boundaries can be addressed separately.

        Attachments

          Activity

            People

            • Assignee:
              ChrisEgerton Chris Egerton
              Reporter:
              ChrisEgerton Chris Egerton
            • Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: