Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-9941

WorkerSinkTask:When a record triggers a RetriableException and the retry is processed successfully, its offset does not commit.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.2.0
    • Component/s: KafkaConnect
    • Labels:
      None

      Description

      When a record triggers a RetriableException and the retry is processed successfully, its offset does not commit.

      The processing process of connect is as follows:

      1: commitOffsets

      2: poll (long timeoutMs)

      3: convertMessages

      4: deliverMessages

      Offset storage:

      1: lastCommittedOffsets

      2: currentOffsets

      3: origOffsets

      Cause of the problem: I need to retry the record

      1.RetriableException from SinkTask:

      2.Pausing partitions

      3.Not returning fetched records for assigned partition since it is no longer fetchable

         3.1.ConsumerRecords<byte[], byte[]> msgs = pollConsumer(timeoutMs);

         3.2.convertMessages(msgs);// msgs is empty

         3.3.origOffsets.clear(); // Record of the retry operation, it's offset has been cleared

      4.The retry operation has completed

          4.1.now commit the offset of the  record of the retry operation, but origOffsets has been cleared.

          4.2.Skipping offset commit, no change since last commit

          4.3.Finished offset commit successfully in 0 ms for sequence number 384: null

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              jiangchuan-java jiangchuan
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: