Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-3260

Increase the granularity of commit for SourceTask

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.0.1
    • Fix Version/s: 0.10.0.0
    • Component/s: KafkaConnect
    • Labels:
      None

      Description

      As of right now when commit is called the developer does not know which messages have been accepted since the last poll. I'm proposing that we extend the SourceTask class to allow records to be committed individually.

          public void commitRecord(SourceRecord record) throws InterruptedException {
              // This space intentionally left blank.
          }
      

      This method could be overridden to receive a SourceRecord during the callback of producer.send. This will give us messages that have been successfully written to Kafka. The developer then has the capability to commit messages to the source individually or in batch.

        Attachments

          Activity

            People

            • Assignee:
              ewencp Ewen Cheslack-Postava
              Reporter:
              jcustenborder Jeremy Custenborder
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: