[CAMEL-20044] camel-kafka - On rejoining consumer group Camel can set offset incorrectly causing messages to be replayed

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.21.0
• Fix Version/s: 3.21.3, 3.22.0, 4.3.0
• Component/s: camel-kafka

Environment

• Rocky Linux 8.7
• Open JDK 11.0.8
• Camel 3.21.0
• Spring Boot 2.7.14
• Strimzi Kafka 0.28.0/3.0.0

• Estimated Complexity: Unknown
• Assignee: Otavio Rodolfo Piske
• Reporter: Mike Barlotta
• Created: 24/Oct/23 15:35
• Updated: 10/Jan/24 09:24
• Resolved: 20/Nov/23 16:15

Description

Reproducing (intermittent):

• Configure camel kafka consumer with following:
  • autoCommitEnable = false
  • allowManualCommit = true
  • autoOffsetReset = earliest
  • maxPollRecords = 1
  • breakOnFirstError = true
• Produce a series of records to kafka record to both partitions.
• Throw an exception that is unhandled
• commit the offset in the onException block

Expected behavior:

• Application should consume the record 1 more time, then move on to the next offset in the partition

Actual behavior:

• Application will often work. Occasionally will use the offset from another partition and assign that to the partition where the record failed. This can then result in the consumer replaying messages instead of moving forward.

I put together a sample that can recreate the error. However, given its intermittent nature it may not fail on each run. I have included the logs from 3 different runs on my laptop from this test. Two of them show the error occurring. One of the them has a successful run. I have also provided more details in the README. 

• https://github.com/CodeSmell/CamelKafkaOffset

This seems related to other issues with how Camel processes the breakOnFirstError attribute. 

• CAMEL-14935
• CAMEL-17925
• CAMEL-18350
• CAMEL-19894

Issue Links

• causes: CAMEL-20089 camel-kafka: make breakOnFirstError more flexible
• is related to: CAMEL-14935 KafkaConsumer commits old offset values in a failure scenario causing message replays and offset reset error
• is related to: CAMEL-17925 camel-kafka: "breakOnFirstError" option is not respected
• is related to: CAMEL-18350 camel-kafka: enabling "breakOnFirstError" causes camel to reconsume all records on error
• is related to: CAMEL-18760 camel-kafka - Issue using ThrottlingExceptionRoutePolicy with Kafka consumer
• is related to: CAMEL-19894 camel-kafka: enabling "breakOnFirstError" causes to skip records on exception
• relates to: CAMEL-20235 camel-kafka: consolidate commit management behavior between component and endpoint
• links to: GitHub Pull Request #11920
• links to: GitHub Pull Request #11935
• links to: GitHub Pull Request #11959
• links to: GitHub Pull Request #11960
• links to: GitHub Pull Request #11970
• links to: GitHub Pull Request #12002
• links to: GitHub Pull Request #12003
• links to: GitHub Pull Request #12004
• links to: GitHub Pull Request #12108
• links to: GitHub Pull Request #12723