[KAFKA-14727] Connect EOS mode should periodically call task commit

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0, 3.4.0, 3.3.1, 3.3.2
• Fix Version/s: 3.5.0, 3.4.1, 3.3.3
• Component/s: connect
• Assignee: Greg Harris
• Reporter: Greg Harris
• Created: 16/Feb/23 17:11
• Updated: 17/Feb/23 17:22
• Resolved: 16/Feb/23 23:52

Description

In non-EOS mode, there is a background thread which periodically commits offsets for a task. If this thread does not have resources to flush on the framework side (records, or offsets) it still calls the task's commit() method to update the internal state of the task.

In EOS mode, there is no background thread, and all offset commits are performed on the main task thread in response to sending records to Kafka. This has the effect of only triggering the task's commit() method when there are records to send to Kafka, which is different than non-EOS mode.

In order to bring the two modes into better alignment, and allow tasks reliant on the non-EOS empty commit() behavior to work in EOS mode out-of-the-box, EOS mode should provide offset commits periodically for tasks which do not produce records.

Issue Links

• Discovered while testing: KAFKA-12468 Initial offsets are copied from source to target cluster
• links to: GitHub Pull Request #13262