Major Description
The Javadocs for SourceTask::commit state that the method should:
Commit the offsets, up to the offsets that have been returned by poll().
However, this is obviously incorrect given how the Connect runtime (when not configured with exactly-once support for source connectors) performs polling and offset commits on separate threads. There's also some extensive discussion on the semantics of that method in KAFKA-5716 where it's made clear that altering the behavior of the runtime to align with the documented semantics of that method is not a viable option.
We should update the Javadocs for this method to state that it does not have anything to do with the offsets returned from SourceTask:poll and is instead just a general, periodically-invoked hook to let the task know that an offset commit has taken place (but with no guarantees as to which offsets have been committed and which ones correspond to still-in-flight records).
Attachments
Issue Links
- is caused by
-
KAFKA-5716 Connect: When SourceTask.commit it is possible not everthing from SourceTask.poll has been sent
-
- Patch Available
-
- links to