[HDFS-16940] Provide HTTP API health endpoints to simplify monitoring

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.4.0, 3.3.9
• Component/s: hdfs
• Target Version/s: 3.3.9, 3.5.0
• Assignee: Steve Vaughan
• Reporter: Steve Vaughan
• Created: 03/Mar/23 14:51
• Updated: 04/Jan/24 08:05

Description

Provide HTTP endpoints that report the current status of the API server when running in dynamic environments.  Using Kubernetes as an example, this would include `/livez` and `/readyz` endpoints.  These endpoints would be distinct, covering specific cases:

• `livez` would indicate if the server is up and running.  A service can be up but not ready to to cover it's assigned duties.  As an example, a JournalNode could be up, but unformatted or unsynchronized.
• `readyz` would indicate that the server ready.  Following the same example, once a JournalNode is synchronized it can act as part of the  JournalNode quorum, supporting other services.

This type of service would increase stability by enabling rolling updates that maintain a quorum of ready participating service instances.