Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-3821

Allow Kafka Connect source tasks to produce offset without writing to topics

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.9.0.1
    • Fix Version/s: None
    • Component/s: KafkaConnect
    • Labels:

      Description

      Provide a way for a SourceTask implementation to record a new offset for a given partition without necessarily writing a source record to a topic.

      Consider a connector task that uses the same offset when producing an unknown number of SourceRecord objects (e.g., it is taking a snapshot of a database). Once the task completes those records, the connector wants to update the offsets (e.g., the snapshot is complete) but has no more records to be written to a topic. With this change, the task could simply supply an updated offset.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                rhauch Randall Hauch
              • Votes:
                7 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated: