[CAMEL-19491] Failing healthcheck on aws2-* polling consumers causes readiness check to be stuck

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.20.5
• Fix Version/s: 3.20.7, 3.21.1, 3.22.0, 4.0.0
• Component/s: camel-aws2
• Labels: easy help-wanted
• Estimated Complexity: Novice
• Assignee: Claus Ibsen
• Reporter: Simon Rasmussen
• Created: 23/Jun/23 09:57
• Updated: 27/Jul/23 16:10
• Resolved: 27/Jul/23 16:10

Description

update: First observed on SQS component, but other aws polling consumers affected as well.

If a Sqs2ConsumerHealthCheck returns DOWN, then it will not recover back to UP, despite the consumer polling messages and processing them successfully.

Detected on SQS, but likely all aws2 components are affected by this.

We have experienced this a few times in production now (on various camel 2.20.x versions), including 3.20.5.

Actual output from our readiness check:

{"status":"DOWN","components":{"camelHealth":{"status":"DOWN","details":{"name":"camel-health-check","consumer:queue_name":"DOWN"}},"db":{"status":"UP","details":{"database":"MariaDB","validationQuery":"isValid()"}},"diskSpace":{"status":"UP","details":{"total":64411906048,"free":13194371072,"threshold":10485760,"exists":true}},"ping":{"status":"UP"},"readinessState":{"status":"UP"}}}

Notice how the health check prefix is absent: aws2-sqs-consumer-

I noticed that the tests of this functionality are manually plumbing the setup.

I also see that Sqs2ConsumerHealthCheck extends AbstractHealthCheck, but shouldn't this be ConsumerHealthCheck instead?

My availability does not allow for attempting to fix this myself, thus I've just created this ticket for now, maybe someone else is up for grabbing it.

Issue Links

• is related to: CAMEL-19281 Aws2- healthchecks not closing resources for awsClient
• relates to: CAMEL-16975 Components should be able to provide custom HealthCheck
• links to: GitHub Pull Request #10847
• links to: GitHub Pull Request #10860