[CASSANDRA-19666] Split the metrics collection into smaller groups

• Type: Improvement
• Status: Patch Available
• Priority: Urgent
• Resolution: Unresolved
• Fix Version/s: 5.x
• Component/s: Observability/Metrics
• Change Category: Performance
• Complexity: Normal
• Platform: All

Test and Documentation Plan

Run the tests, the benchmarks are also required.

• Assignee: Maxim Muzafarov
• Reporter: Maxim Muzafarov

Authors

Maxim Muzafarov

• Created: 28/May/24 19:02
• Updated: 10/Oct/24 11:22

Description

Historically, all metrics are stored in a single collection indexed by a unique metric name - ConcurrentMap<String, Metric>. This storage technique doesn't offer any advantages over other options we might have below, and causes some performance and usability issues when we need to traverse this huge collection while filtering the metrics.

The following possible solutions could be used: 

• Use https://www.javadoc.io/doc/io.dropwizard.metrics/metrics-core/4.0.2/com/codahale/metrics/SharedMetricRegistries.html , which provides a way to store metrics in their dedicated groups;
• Implement a hierarchical storage structure, based on the knowledge of how the metric key is formed. For example, https://the-asf.slack.com/archives/CK23JSY2K/p1716568216701789?thread_ts=1716324924.466269&cid=CK23JSY2K

Each of the above options assumes, that the metric groups are statically registered and known at the time a node starts. This may also be an option to consider for dynamically registering metric groups and in turn registering corresponding virtual tables and JmxMbean. In the latter case, we'll need to implement the notification mechanism for the drivers, as they are not currently notified about newly created virtual tables.

Issue Links

• links to: GitHub Pull Request #3452