Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
This is a follow-up of KAFKA-14294.
Today we have several flags to determine if KS need to execute a commit: 1) task-level "commitNeeded" which is set whenever process() or punctuator() is called, 2) if there are input topic offsets to commit, retrieved from the "task.prepareCommit()", 3) the "transactionInFlight" flag from producer as a fix of KAFKA-14294 (this subsumes the first "commitNeeded" functionality).
Given that we are still having EOS-v1, cleanup this would be a bit complex. But after the deprecated EOS-V1 is removed, we can cleanup those controls since for any commit cases, we would need to commit all tasks anyways whereas in EOS-v1, we would commit probably a subset of tasks since they are done by different producers and hence different txns.
A quick thought is the following:
1) We would not need the per-task "commitNeeded" anymore.
2) We would maintain a single "commitNeeded" flag on the task-executor, hence on the thread level. It is set whenever `process()` or `punctuator` is called.
3) Whenever we need to commit, either a) periodically, b) upon revocation, c) upon user request, we simply check that flag, and if necessary commit all tasks and reset the flag.
Attachments
Issue Links
- is blocked by
-
KAFKA-12689 Remove deprecated EOS configs
- Resolved