[ZOOKEEPER-4818] Export JVM heap metrics in ServerMetrics

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: metric system
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Andrew Kyle Purtell
• Created: 25/Mar/24 19:36
• Updated: 09/Jul/24 16:16

Description

A metric for JVM heap occupancy is not included in ServerMetrics.

According to https://zookeeper.apache.org/doc/current/zookeeperMonitor.html the recommended practice is for someone to enable the PrometheusMetricsProvider and the Prometheus base class upon which that provider is based does export that information. See https://zookeeper.apache.org/doc/current/zookeeperMonitor.html . The example provided for alerting on heap utilization there is:

  - alert: JvmMemoryFillingUp
    expr: jvm_memory_bytes_used / jvm_memory_bytes_max{area="heap"} > 0.8
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "JVM memory filling up (instance {{ $labels.instance }})"
      description: "JVM memory is filling up (> 80%)\n labels: {{ $labels }}  value = {{ $value }}\n"

where jvm_memory_bytes_used and jvm_memory_bytes_max are provided by a Prometheus base class.

Where PrometheusMetricsProvider is the right choice that's good enough but where the ServerMetrics information is consumed in an alternate way, by 4-letter-word scraping, or by JMX, ServerMetrics should provide the same information. jvm_memory_bytes_used and jvm_memory_bytes_max (presuming heap) are reasonable names. An alternative could be to calculate the heap occupancy and provide that as a percentage, either an integer in the range 0 - 100 or floating point value in the range 0.0 - 1.0. 

There is some precedent for exporting JVM metrics in ServerMetrics from ZOOKEEPER-3845 . 

Issue Links

• is related to: ZOOKEEPER-3845 Add metric JVM_PAUSE_TIME
• links to: GitHub Pull Request #2170