Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-2786

Only invoke SinkTask onPartitionsRevoked and commitOffsets after task has fully started

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • None
    • 0.9.0.0
    • connect
    • None

    Description

      Currently we're invoking task.onPartitionsRevoked and commitOffsets on any consumer onPartitionsRevoked callback. Since this happens even on the first rebalance (which is somewhat unexpected given that partitions cannot actually be revoked at that point), we will not be fully setup in the SinkTask and this can throw exceptions.

      It turns out this is easy to miss for two reasons. First, the code is likely to throw an exception, but the consumer catches this and handles it. We end up not missing any important steps as a result. Second, the logging was a bit screwed up and we weren't getting the full exception stacktrace, just the toString(), so it was easy to miss that there was a problem.

      Attachments

        Activity

          People

            ewencp Ewen Cheslack-Postava
            ewencp Ewen Cheslack-Postava
            Gwen Shapira Gwen Shapira
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: