Uploaded image for project: 'Camel'
  1. Camel
  2. CAMEL-20044

camel-kafka - On rejoining consumer group Camel can set offset incorrectly causing messages to be replayed



    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.21.0
    • 3.21.3, 3.22.0, 4.3.0
    • camel-kafka
    • None
      • Rocky Linux 8.7
      • Open JDK 11.0.8
      • Camel 3.21.0
      • Spring Boot 2.7.14
      • Strimzi Kafka 0.28.0/3.0.0
    • Unknown


      Reproducing (intermittent):

      • Configure camel kafka consumer with following:
        • autoCommitEnable = false
        • allowManualCommit = true
        • autoOffsetReset = earliest
        • maxPollRecords = 1
        • breakOnFirstError = true
      • Produce a series of records to kafka record to both partitions.
      • Throw an exception that is unhandled
      • commit the offset in the onException block

      Expected behavior:

      • Application should consume the record 1 more time, then move on to the next offset in the partition

      Actual behavior:

      • Application will often work. Occasionally will use the offset from another partition and assign that to the partition where the record failed. This can then result in the consumer replaying messages instead of moving forward.

      I put together a sample that can recreate the error. However, given its intermittent nature it may not fail on each run. I have included the logs from 3 different runs on my laptop from this test. Two of them show the error occurring. One of the them has a successful run. I have also provided more details in the README. 

      This seems related to other issues with how Camel processes the breakOnFirstError attribute. 


        1. camel-kafka-offset.11-06-2023.log
          173 kB
          Mike Barlotta

        Issue Links



              orpiske Otavio Rodolfo Piske
              g1antfan Mike Barlotta
              2 Vote for this issue
              4 Start watching this issue