Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-2786

Only invoke SinkTask onPartitionsRevoked and commitOffsets after task has fully started

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.9.0.0
    • Component/s: KafkaConnect
    • Labels:
      None

      Description

      Currently we're invoking task.onPartitionsRevoked and commitOffsets on any consumer onPartitionsRevoked callback. Since this happens even on the first rebalance (which is somewhat unexpected given that partitions cannot actually be revoked at that point), we will not be fully setup in the SinkTask and this can throw exceptions.

      It turns out this is easy to miss for two reasons. First, the code is likely to throw an exception, but the consumer catches this and handles it. We end up not missing any important steps as a result. Second, the logging was a bit screwed up and we weren't getting the full exception stacktrace, just the toString(), so it was easy to miss that there was a problem.

        Attachments

          Activity

            People

            • Assignee:
              ewencp Ewen Cheslack-Postava
              Reporter:
              ewencp Ewen Cheslack-Postava
              Reviewer:
              Gwen Shapira
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: