[SOLR-14210] Add replica state option for HealthCheckHandler

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 8.5
• Fix Version/s: 8.6
• Assignee: Jan Høydahl
• Reporter: Houston Putman
• Created: 23/Jan/20 16:25
• Updated: 15/Jul/20 15:09
• Resolved: 14/Apr/20 15:12

Description

Background

As was brought up in SOLR-13055, in order to run Solr in a more cloud-native way, we need some additional features around node-level healthchecks.

Like in Kubernetes we need 'liveliness' and 'readiness' probe explained in https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/n determine if a node is live and ready to serve live traffic.

 

However there are issues around kubernetes managing it's own rolling restarts. With the current healthcheck setup, it's easy to envision a scenario in which Solr reports itself as "healthy" when all of its replicas are actually recovering. Therefore kubernetes, seeing a healthy pod would then go and restart the next Solr node. This can happen until all replicas are "recovering" and none are healthy. (maybe the last one restarted will be "down", but still there are no "active" replicas)

Proposal

I propose we make an additional healthcheck handler that returns whether all replicas hosted by that Solr node are healthy and "active". That way we will be able to use the default kubernetes rolling restart logic with Solr.

To add on to Jan's point here, this handler should be more friendly for other Content-Types and should use bettter HTTP response statuses.

Issue Links

• is part of: SOLR-13055 Introduce check to determine "liveliness" of a Solr node
• links to: GitHub Pull Request #1387