Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-3850

WorkerSinkTask should retry commits if woken up during rebalance or shutdown

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.0.0
    • Fix Version/s: 0.10.0.1, 0.10.1.0
    • Component/s: KafkaConnect
    • Labels:
      None

      Description

      We use consumer.wakeup() to interrupt long polls when we need to pause/resume partitions and when we shutdown sink tasks. The resulting WakeupException could be raised from the synchronous commit which we use in between rebalances and on shutdown. Since we don't currently catch this exception, we can fail to commit offsets, which typically results in duplicates. To fix this problem, we should catch the exception, retry the commit, and then rethrow it.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hachikuji closed the pull request at:

          https://github.com/apache/kafka/pull/1511

          Show
          githubbot ASF GitHub Bot added a comment - Github user hachikuji closed the pull request at: https://github.com/apache/kafka/pull/1511
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user hachikuji reopened a pull request:

          https://github.com/apache/kafka/pull/1511

          KAFKA-3850: WorkerSinkTask commit prior to rebalance should be retried on wakeup

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/hachikuji/kafka retry-commit-on-wakeup-in-sinks

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/kafka/pull/1511.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #1511


          commit 0e2e4568325a52611f67e172688e2ade86e8bdf5
          Author: Jason Gustafson <jason@confluent.io>
          Date: 2016-06-16T00:40:32Z

          KAFKA-??: WorkerSinkTask commit prior to rebalance should be retried on wakeup


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user hachikuji reopened a pull request: https://github.com/apache/kafka/pull/1511 KAFKA-3850 : WorkerSinkTask commit prior to rebalance should be retried on wakeup You can merge this pull request into a Git repository by running: $ git pull https://github.com/hachikuji/kafka retry-commit-on-wakeup-in-sinks Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1511.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1511 commit 0e2e4568325a52611f67e172688e2ade86e8bdf5 Author: Jason Gustafson <jason@confluent.io> Date: 2016-06-16T00:40:32Z KAFKA-??: WorkerSinkTask commit prior to rebalance should be retried on wakeup
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/kafka/pull/1511

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1511
          Hide
          ewencp Ewen Cheslack-Postava added a comment -

          Issue resolved by pull request 1511
          https://github.com/apache/kafka/pull/1511

          Show
          ewencp Ewen Cheslack-Postava added a comment - Issue resolved by pull request 1511 https://github.com/apache/kafka/pull/1511

            People

            • Assignee:
              hachikuji Jason Gustafson
              Reporter:
              hachikuji Jason Gustafson
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development