Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-14641

Cleanup CommitNeeded after EOS-V1 is removed

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • streams
    • None

    Description

      This is a follow-up of KAFKA-14294.

      Today we have several flags to determine if KS need to execute a commit: 1) task-level "commitNeeded" which is set whenever process() or punctuator() is called, 2) if there are input topic offsets to commit, retrieved from the "task.prepareCommit()", 3) the "transactionInFlight" flag from producer as a fix of KAFKA-14294 (this subsumes the first "commitNeeded" functionality).

      Given that we are still having EOS-v1, cleanup this would be a bit complex. But after the deprecated EOS-V1 is removed, we can cleanup those controls since for any commit cases, we would need to commit all tasks anyways whereas in EOS-v1, we would commit probably a subset of tasks since they are done by different producers and hence different txns.

      A quick thought is the following:

      1) We would not need the per-task "commitNeeded" anymore.
      2) We would maintain a single "commitNeeded" flag on the task-executor, hence on the thread level. It is set whenever `process()` or `punctuator` is called.
      3) Whenever we need to commit, either a) periodically, b) upon revocation, c) upon user request, we simply check that flag, and if necessary commit all tasks and reset the flag.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              guozhang Guozhang Wang
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: