Uploaded image for project: 'Camel'
  1. Camel
  2. CAMEL-18759

camel-kafka - When Kafka pausable consumer is resumed, it reads all the messages from the earliest offset

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.18.3
    • 3.22.0, 4.0.1, 4.1.0
    • camel-kafka
    • None
    • Unknown

    Description

      Hi,

      I'm trying to add the circuit breaker pattern to a Kafka consumer. In short, I want to suspend the consumer when an exception happens due to an outage in downstream systems and resume the consumer when downstream systems are up again. As a reference I used this example from the Camel documentation: https://camel.apache.org/components/3.18.x/eips/resume-strategies.html#_pausable_consumers_api (coming from this test: https://github.com/apache/camel/blob/2c01082032346753cc6621ab4530b0969f12bac7/components/camel-kafka/src/test/java/org/apache/camel/component/kafka/integration/pause/KafkaPausableConsumerCircuitBreakerIT.java)
      However, I noticed that every time the consumer is resumed it starts reading the messages from the earliest offset, and this causes a redelivery of all the messages stored in the partition; as far as I can tell it happens because KafkaConsumerListener#afterProcess invokes consumer.seekToBeginning.

      I've created a reproducer here: https://github.com/lorenzobenvenuti/camel-kafka-pausable-bug

      Thanks,

      lorenzo

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            orpiske Otavio Rodolfo Piske
            lbenvenu Lorenzo Benvenuti
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment