Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-10188

Sink task preCommit method gets called after task is stopped

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: KafkaConnect
    • Labels:
      None

      Description

      When the final cleanup for a sink task is initiated, the framework first calls stop() on the task, and then closes the consumer for the task. Closing the consumer has the side effect of triggering the onPartitionsRevoked method of its ConsumerRebalanceListener, which in turn causes the framework to call WorkerSinkTask::closePartitions, which in turn calls WorkerSinkTask::commitOffsets, and finally, in turn calls SinkTask:preCommit.

      Calling SinkTask:preCommit after SinkTask::stop is likely to cause errors with tasks as they should be performing resource cleanup during stop, and the current documentation on the SinkTask lifecycle makes no mention of anything happening after tasks are stopped.

       

      The framework already ensures that offsets are committed for tasks after the last call to SinkTask:put has been made, so the offset commit after SinkTask::stop has already been invoked can and should be removed with no compromise of existing delivery guarantees provided by the framework.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ChrisEgerton Chris Egerton
                Reporter:
                ChrisEgerton Chris Egerton
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated: