Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-5731

Connect WorkerSinkTask out of order offset commit can lead to inconsistent state

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.10.2.2, 0.11.0.1
    • Component/s: KafkaConnect
    • Labels:
      None

      Description

      In Connect's WorkerSinkTask, we do sequence number validation to ensure that offset commits are handled in the right order (https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L199).

      Unfortunately, for asynchronous commits, the lastCommittedOffsets field is overridden regardless of this sequence check as long as the response had no error (https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L284):

                  OffsetCommitCallback cb = new OffsetCommitCallback() {
                      @Override
                      public void onComplete(Map<TopicPartition, OffsetAndMetadata> offsets, Exception error) {
                          if (error == null) {
                              lastCommittedOffsets = offsets;
                          }
                          onCommitCompleted(error, seqno);
                      }
                  };
      

      Hence if we get an out of order commit, then the internal state will be inconsistent. To fix this, we should only override lastCommittedOffsets after sequence validation as part of the onCommitCompleted(...) method.

        Attachments

          Activity

            People

            • Assignee:
              rhauch Randall Hauch
              Reporter:
              hachikuji Jason Gustafson
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: