Major Description
It is desirable to have, in addition to the time-based flush interval, volume or size-based commits. E.g. a sink connector which is buffering in terms of number of records may want to request a flush when the buffer is full, or when sufficient amount of data has been buffered in a file.
Having a method like say requestFlush() on the SinkTaskContext would allow for connectors to have flexible policies around flushes. This would be in addition to the time interval based flushes that are controlled with offset.flush.interval.ms, for which the clock should be reset when any kind of flush happens.
We should probably also support requesting flushes via the SourceTaskContext for consistency though a use-case doesn't come to mind off the bat.
Attachments
Issue Links
- contains
-
KAFKA-3462 Allow SinkTasks to disable consumer offset commit
-
- Resolved
-
- links to
- mentioned in
-
Page Loading...