[HDDS-9168] Improve Ozone metrics

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Zita Dombi
• Reporter: Zita Dombi
• Created: 14/Aug/23 13:51
• Updated: 14/Aug/23 13:51

Description

Currently Ozone has some metrics exposed to jmx about the state of the cluster, but it could be improved. Our goal is to provide more information, e.g.

• number of FSO/OBS buckets
• more information about the storage size (like total, EC, RATIS storage size)
• volume/disk failures detected

This list contains the first thoughts, other ideas are welcomed. These need to be persisted, so in case the cluster is restarted it won't be lost. in HDDS-816 OmMetricsInfo was created, which can be used for this.

Also there are OM metrics that are never populated (like S3 related ones, but there can be others as well). We need to go through the existing metrics and rethink them, if needed fix the usage of them.

Sub-Tasks

• Add number of datanodes, total capacity/used space to SCMNodeMetrics: Resolved
• NumKeys metric not decremented on FSO directory delete: Resolved
• Create repair tool for OM metrics: Resolved
• Add FSO bucket related (count, operations) metrics: Open
• Add separate OM metric for directory count: Open
• Parameterize TestOmMetrics#testKeyOps across bucket layouts: Open
• Use RocksDB key count estimates instead of OM metrics file: Open