[HBASE-21991] Fix MetaMetrics issues - [Race condition, Faulty remove logic], few improvements

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha-1, 2.2.0, 1.4.10, 2.3.0, 2.1.5
• Component/s: Coprocessors, metrics
• Hadoop Flags: Incompatible change

Release Note

The class LossyCounting was unintentionally marked Public but was never intended to be part of our public API. This oversight has been corrected and LossyCounting is now marked as Private and going forward may be subject to additional breaking changes or removal without notice. If you have taken a dependency on this class we recommend cloning it locally into your project before upgrading to this release.

• Assignee: Sakthi
• Reporter: Sakthi
• Created: 05/Mar/19 01:57
• Updated: 07/Oct/19 18:31
• Resolved: 25/May/19 22:17

Description

Here is a list of the issues related to the MetaMetrics implementation:

Bugs:

1. [_Lossy counting for top-k_] Faulty remove logic of non-eligible meters: Under certain conditions, we might end up storing/exposing all the meters rather than top-k-ish
2. MetaMetrics can throw NPE resulting in aborting of the RS because of a Race Condition.

Improvements:

1. With high number of regions in the cluster, exposure of metrics for each region blows up the JMX from ~140 Kbs to 100+ Mbs depending on the number of regions. It's better to use lossy counting to maintain top-k for region metrics as well.
2. As the lossy meters do not represent actual counts, I think, it'll be better to rename the meters to include "lossy" in the name. It would be more informative while monitoring the metrics and there would be less confusion regarding actual counts to lossy counts.

Issue Links

• relates to: HBASE-19722 Meta query statistics metrics source
• relates to: HBASE-23054 Remove synchronization block from MetaTableMetrics and fix LossyCounting algorithm