[CAMEL-18759] camel-kafka - When Kafka pausable consumer is resumed, it reads all the messages from the earliest offset

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.18.3
• Fix Version/s: 3.22.0, 4.0.1, 4.1.0
• Component/s: camel-kafka
• Estimated Complexity: Unknown
• Assignee: Otavio Rodolfo Piske
• Reporter: Lorenzo Benvenuti
• Created: 25/Nov/22 10:04
• Updated: 20/Sep/23 09:27
• Resolved: 20/Sep/23 09:27

Description

Hi,

I'm trying to add the circuit breaker pattern to a Kafka consumer. In short, I want to suspend the consumer when an exception happens due to an outage in downstream systems and resume the consumer when downstream systems are up again. As a reference I used this example from the Camel documentation: https://camel.apache.org/components/3.18.x/eips/resume-strategies.html#_pausable_consumers_api (coming from this test: https://github.com/apache/camel/blob/2c01082032346753cc6621ab4530b0969f12bac7/components/camel-kafka/src/test/java/org/apache/camel/component/kafka/integration/pause/KafkaPausableConsumerCircuitBreakerIT.java)
However, I noticed that every time the consumer is resumed it starts reading the messages from the earliest offset, and this causes a redelivery of all the messages stored in the partition; as far as I can tell it happens because KafkaConsumerListener#afterProcess invokes consumer.seekToBeginning.

I've created a reproducer here: https://github.com/lorenzobenvenuti/camel-kafka-pausable-bug

Thanks,

lorenzo

Issue Links

• relates to: CAMEL-18760 camel-kafka - Issue using ThrottlingExceptionRoutePolicy with Kafka consumer