[HBASE-9774] HBase native metrics and metric collection for coprocessors

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: Coprocessors, metrics
• Hadoop Flags: Reviewed

Release Note

This issue adds two new modules, hbase-metrics and hbase-metrics-api which define and implement the "new" metric system used internally within HBase. These two modules (and some other code in hbase-hadoop2-compat) module are referred as "HBase metrics framework" which is HBase-specific and independent of any other metrics library (including Hadoop metrics2 and dropwizards metrics).

HBase Metrics API (hbase-metrics-api) contains the interface that HBase exposes internally and to third party code (including coprocessors). It is a thin
abstraction over the actual implementation for backwards compatibility guarantees. The metrics API in this hbase-metrics-api module is inspired by the Dropwizard metrics 3.1 API, however, the API is completely independent.

hbase-metrics module contains implementation of the "HBase Metrics API", including MetricRegistry, Counter, Histogram, etc. These are highly concurrent implementations of the Metric interfaces. Metrics in HBase are grouped into different sets (like WAL, RPC, RegionServer, etc). Each group of metrics should be tracked via a MetricRegistry specific to that group.

Historically, HBase has been using Hadoop's Metrics2 framework [3] for collecting and reporting the metrics internally. However, due to the difficultly of dealing with the Metrics2 framework, HBase is moving away from Hadoop's metrics implementation to its custom implementation. The move will happen incrementally, and during the time, both Hadoop Metrics2-based metrics and hbase-metrics module based classes will be in the source code. All new implementations for metrics SHOULD use the new API and framework.

This jira also introduces the metrics API to coprocessor implementations. Coprocessor writes can export custom metrics using the API and have those collected via metrics2 sinks, as well as exported via JMX in regionserver metrics.

More documentation available at: hbase-metrics-api/README.txt

• Assignee: Enis Soztutar
• Reporter: Gary Helmling
• Created: 15/Oct/13 23:05
• Updated: 01/Mar/23 16:26
• Resolved: 25/Jan/17 19:51

Description

It would help provide better visibility into what coprocessors are doing if we provided a way for coprocessors to export their own metrics. The general idea is to:

• extend access to the HBase "metrics bus" down into the coprocessor environments
• coprocessors can then register and increment custom metrics
• coprocessor metrics are then reported along with all others through normal mechanisms

Issue Links

• breaks: HBASE-17964 ensure hbase-metrics-api is included in mapreduce job classpaths
• breaks: HBASE-17622 Add hbase-metrics package to TableMapReduceUtil
• is related to: HBASE-20904 Prometheus metrics http endpoint for monitoring integration
• is related to: HBASE-27681 Refactor Table Latency Metrics
• is related to: HBASE-14996 Some more API cleanup for 2.0
• relates to: HBASE-17536 User documentation for coprocessor metrics
• relates to: HBASE-18060 Backport to branch-1 HBASE-9774 HBase native metrics and metric collection for coprocessors
• relates to: HBASE-14282 Remove metrics2