[CAMEL-15562] repository & strategy to store and retrieve the latest state of a consumer and allow to resume upon restart

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.16.0
• Component/s: camel-core-api, camel-core-engine
• Estimated Complexity: Unknown
• Assignee: Otavio Rodolfo Piske
• Reporter: Luca Burgazzoli
• Created: 21/Sep/20 18:45
• Updated: 08/Mar/22 10:13
• Resolved: 08/Mar/22 10:13

Description

A number of component support resuming operation from a given point in time or index but each component implement its own logic and it is currently not possible to use an external system to store such information.

Because of that, even if the source support resuming from a point, we sometime still need to use an idempotent consumer which is ok but inefficient because the consumer may need to re-process events from the source as it lacks a way to retrieve from where it left.

We should provide a generic mechanism to implement such logic so that:

1. it should be possible to restart a camel application without having to re-process events from the source even without the need of an idempotent repository (for sources that support resuming)
2. in case of sources that support partitioning, it should be possible to consume data from an elastic pool of camel consumers as the information about the consumer state is stored in an external repository so the system can rebalance the load anc new consumer can start where others have left

InProgressRepository

This repository is similar to the IndempotentRepository except it should allow to query the status of the entries.

InProgressStrategy

A simple strategy that automatically update the InProgressRepository according to some headers like:

1. CamelInProgressID: the key of the event
2. CamelInProgressValue: the value (opaque)
3. CamelInProgressStatus: the status of the consumer for the given key (i.e. could be InProgress when the consumer is processing a file and become Done when the file is fully processed)

We should also implement a mechanism to be able to consistently determine the "partition key" according to the active consumers.

Issue Links

• is related to: CAMEL-15955 camel-core - API and SPI for Camel components to resume from offset
• relates to: CAMEL-17026 camel-kafka: add resume support on the producer
• relates to: CAMEL-17051 camel-kafka: Need the ability to pause kafka consumer
• relates to: CAMEL-17045 camel-file: add support for the resume strategy
• relates to: CAMEL-17078 camel-file: add support for the resume from offset strategy
• relates to: CAMEL-17108 camel-couchdb: investigate adding resume strategy
• relates to: CAMEL-17117 eip - Add Resume from Offset as an EIP to the docs
• relates to: CAMEL-17186 camel-aws2-kinesis: add support for resume strategies
• relates to: CAMEL-17733 camel-resume-api: create a delegating strategy
• links to: GitHub Pull Request #6793
• links to: GitHub Pull Request #6899
• links to: GitHub Pull Request #7085