[CAMEL-18350] camel-kafka: enabling "breakOnFirstError" causes camel to reconsume all records on error

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.18.0
• Fix Version/s: 3.14.6, 3.18.3, 3.20.0
• Component/s: camel-kafka
• Estimated Complexity: Unknown
• Regression: Regression
• Assignee: Karen Lease
• Reporter: Espen Andreassen
• Created: 04/Aug/22 13:04
• Updated: 07/Nov/23 10:08
• Resolved: 29/Sep/22 15:41

Description

Reproducing:

• Configure camel kafka consumer with with "breakOnFirstError" = "true"
• Produce a series of records to kafka record consumed by application
• Ensure offset is commited
• Produce more records
• Trigger an error when processing one of the records

Expected behavior:

• Application should only reconsume records after the last committed offset

Actual behavior:

• Application reconsumes all records on topic

Whether the erroneous behavior is triggered seem to be dependent on the offset the failing exchange has internally in the series of records in the poll. 

 

I've created a project on github with a failing test that reproduces the issue: https://github.com/espeandr/camel-kafka-incorrectly-reconsumes-entire-topic-demo. I haven't been able to find a pattern to which "message offset" in the poll causes the error. In the test I've reproduced the error by forcing a max poll size as this simplifies the test setup. The issue is also reproducible without overriding maxPollSize.

I suspect that offset = 0 is committed when breaking out in the erroneous cases.

Issue Links

• causes: CAMEL-20089 camel-kafka: make breakOnFirstError more flexible
• relates to: CAMEL-20044 camel-kafka - On rejoining consumer group Camel can set offset incorrectly causing messages to be replayed
• links to: GitHub Pull Request #8447
• links to: GitHub Pull Request #8451