Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-8229

Connect Sink Task updates nextCommit when commitRequest is true

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.3.0, 2.2.2
    • Component/s: KafkaConnect
    • Labels:
      None

      Description

      Today, when a WorkerSinkTask uses context.requestCommit(), the next call to iteration will cause the commit to happen. As part of the commit execution it will also change the nextCommit milliseconds.

      This creates some weird behaviors when a SinkTask calls context.requestCommit multiple times. In our case, we were calling requestCommit when the number of kafka records we processed exceed a threshold. This resulted in the nextCommit being several days in the future and caused it to only commit when the record threshold was reached.

      We expected the task to commit when the record threshold was reached OR when the timer went off.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                ScottReynolds Scott Reynolds
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: