Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-3821

Allow Kafka Connect source tasks to produce offset without writing to topics

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 0.9.0.1
    • None
    • connect

    Description

      Provide a way for a SourceTask implementation to record a new offset for a given partition without necessarily writing a source record to a topic.

      Consider a connector task that uses the same offset when producing an unknown number of SourceRecord objects (e.g., it is taking a snapshot of a database). Once the task completes those records, the connector wants to update the offsets (e.g., the snapshot is complete) but has no more records to be written to a topic. With this change, the task could simply supply an updated offset.

      Attachments

        Issue Links

          Activity

            People

              rhauch Randall Hauch
              rhauch Randall Hauch
              Votes:
              4 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated: