[CAMEL-17051] camel-kafka: Need the ability to pause kafka consumer

• Type: New Feature
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.12.0
• Fix Version/s: 3.17.0
• Component/s: camel-kafka
• Estimated Complexity: Unknown
• Assignee: Otavio Rodolfo Piske
• Reporter: Zachariah Young
• Created: 07/Oct/21 13:31
• Updated: 18/May/22 15:02
• Resolved: 13/Apr/22 09:05

Description

When handling transient errors that have exhausted retry limits, sometimes pausing the Kafka consumer is the best way to handle an outage.  A great example of this would be the Facebook outage that happen this week.

One might suggest that stopping the consumer work would for this use case.  This would of course work but would require a manual restart.  Also, you want to try to avoid a rebalance of the consumer.

I would like to see a new strategy be created that would allow for pause and resume to be configured.

Ideally, the resume could be configured based on a health check.

 

Example code: https://github.com/jeanlouisboudart/retriable-consumer/blob/master/consumer/src/main/java/com/sample/InfiniteRetriesConsumer.java

 

Issue Links

• is related to: CAMEL-15562 repository & strategy to store and retrieve the latest state of a consumer and allow to resume upon restart
• is related to: CAMEL-17762 camel-resume-api: improvements and additional support
• is related to: CAMEL-18125 camel-resume-api: improvements and additional support
• relates to: CAMEL-18003 camel-kafka: manual pause of consumer causes ConcurrentModificationException
• links to: GitHub Pull Request #526
• links to: GitHub Pull Request #7297
• links to: GitHub Pull Request #7387
• links to: GitHub Pull Request #7474
• links to: GitHub Pull Request #7475