Major Description
There is a race condition in the Hbase and Phoenix metrics handling.
Hbase metrics is supposed to run in the daemon processes, while Phoenix metrics is supposed to run on the phoenix client side. create its own DefaultMetrics object on the client side, and add its metrics to the HBase DefaultMetrics object on the RS side.
On an RS both HBase and Phoenix tries to register its metrics implementation into DefaultMetricsSystem.
On the happy path, when Phoenix tries to register its metrics on the RS, it won't have any effect, because HBase has already registered an active Metrics implementation.
However, JmxCacheBuster stops the Hbase metrics system for ~500ms every five seconds, and if Phoenix tries to initialize its metrics in this period, then it will succeed in initializing its Metrics object, and will overwrite theDefaultMetricsSystem object initialized by HBase.
This disables the HBase metrics system for the RS process.
Attachments
Issue Links
- links to