[CAMEL-20089] camel-kafka: make breakOnFirstError more flexible

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.x
• Component/s: camel-kafka
• Estimated Complexity: Unknown
• Assignee: Otavio Rodolfo Piske
• Reporter: Otavio Rodolfo Piske
• Created: 07/Nov/23 10:05
• Updated: 03/Oct/24 17:05
• Resolved: 03/Oct/24 17:05

Description

We have a very high incidence of problems in the camel-kafka component that are related to the breakOnFirstError flag.

Looking at the tickets related to this issue it seems to me that different uses have different expectations about how the component should behave in terms of polling, rolling back, and/or future processing.

In short: this flag is leading to a lot of confusion and we should investigate how we can flexibilize the behavior of the Kafka component under those circumstances and let the users choose more freely the behavior that is suitable to their needs.

Issue Links

• is caused by: CAMEL-14935 KafkaConsumer commits old offset values in a failure scenario causing message replays and offset reset error
• is caused by: CAMEL-20044 camel-kafka - On rejoining consumer group Camel can set offset incorrectly causing messages to be replayed
• is caused by: CAMEL-17925 camel-kafka: "breakOnFirstError" option is not respected
• is caused by: CAMEL-18350 camel-kafka: enabling "breakOnFirstError" causes camel to reconsume all records on error
• is caused by: CAMEL-18760 camel-kafka - Issue using ThrottlingExceptionRoutePolicy with Kafka consumer
• is caused by: CAMEL-19894 camel-kafka: enabling "breakOnFirstError" causes to skip records on exception