Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-4161

Decouple flush and offset commits

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.10.2.0
    • Component/s: KafkaConnect
    • Labels:

      Description

      It is desirable to have, in addition to the time-based flush interval, volume or size-based commits. E.g. a sink connector which is buffering in terms of number of records may want to request a flush when the buffer is full, or when sufficient amount of data has been buffered in a file.

      Having a method like say requestFlush() on the SinkTaskContext would allow for connectors to have flexible policies around flushes. This would be in addition to the time interval based flushes that are controlled with offset.flush.interval.ms, for which the clock should be reset when any kind of flush happens.

      We should probably also support requesting flushes via the SourceTaskContext for consistency though a use-case doesn't come to mind off the bat.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                shikhar Shikhar Bhushan
                Reporter:
                shikhar Shikhar Bhushan
              • Votes:
                1 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: