[HBASE-12911] Client-side metrics

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.0, 1.3.0, 0.98.16, 2.0.0
• Component/s: Client, Operability, Performance

Release Note

Introduces collection and reporting of various client-perceived metrics. Metrics are exposed via JMX under "org.apache.hadoop.hbase.client.MetricsConnection". Metrics are scoped according to connection instance, so multiple connection objects (ie, to different clusters) will report their metrics separately. Metrics are disabled by default, must be enabled by configuring "hbase.client.metrics.enable=true".

• Assignee: Nick Dimiduk
• Reporter: Nick Dimiduk
• Created: 23/Jan/15 19:21
• Updated: 19/Dec/22 22:21
• Resolved: 09/Oct/15 17:15

Description

There's very little visibility into the hbase client. Folks who care to add some kind of metrics collection end up wrapping Table method invocations with System.currentTimeMillis(). For a crude example of this, have a look at what I did in PerformanceEvaluation for exposing requests latencies up to IntegrationTestRegionReplicaPerf. The client is quite complex, there's a lot going on under the hood that is impossible to see right now without a profiler. Being a crucial part of the performance of this distributed system, we should have deeper visibility into the client's function.

I'm not sure that wiring into the hadoop metrics system is the right choice because the client is often embedded as a library in a user's application. We should have integration with our metrics tools so that, i.e., a client embedded in a coprocessor can report metrics through the usual RS channels, or a client used in a MR job can do the same.

I would propose an interface-based system with pluggable implementations. Out of the box we'd include a hadoop-metrics implementation and one other, possibly dropwizard/metrics.

Thoughts?

Issue Links

• is depended upon by: HBASE-14381 Expose client-side metrics via Connection API
• relates to: HBASE-10791 Add integration test to demonstrate performance improvement
• relates to: HBASE-5162 Basic client pushback mechanism
• relates to: HBASE-11062 hbtop
• relates to: PHOENIX-1452 Add Phoenix client-side logging and capture resource utilization metrics
• relates to: PHOENIX-1819 Build a framework to capture and report phoenix client side request level metrics
• relates to: HBASE-14692 IntegrationTestRegionReplicaPerf should consume client side metrics rather than self-monitor
• relates to: ACCUMULO-3908 Instrument client API for metrics
• relates to: HBASE-15242 Client metrics for retries and timeouts
• relates to: HBASE-27540 Client metrics for success/failure counts.
• relates to: HBASE-4145 Provide metrics for hbase client
• relates to: HBASE-13124 Option to collect latencies for individual requests
• relates to: HBASE-14693 Add client-side metrics for received pushback signals
• relates to: HBASE-3869 RegionServer metrics - add read and write byte-transfer statistics
• relates to: HBASE-7169 Adding latency counters to HTableMultiplexer
• relates to: HBASE-1095 Client historian
• relates to: HBASE-14534 Bump yammer/coda/dropwizard metrics dependency version
• links to: https://reviews.apache.org/r/38104/
• links to: RB for Yammer Metrics patch