[SOLR-4735] Improve Solr metrics reporting

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 6.4, 7.0
• Component/s: metrics
• Assignee: Andrzej Bialecki
• Reporter: Alan Woodward
• Created: 18/Apr/13 11:20
• Updated: 02/Dec/20 13:35
• Resolved: 27/Dec/16 15:31

Description

Following on from a discussion on the mailing list:
http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+

It would be good to make Solr play more nicely with existing devops monitoring systems, such as Graphite or Ganglia. Stats monitoring at the moment is poll-only, either via JMX or through the admin stats page. I'd like to refactor things a bit to make this more pluggable.

This patch is a start. It adds a new interface, InstrumentedBean, which extends SolrInfoMBean to return a [Metrics MetricRegistry, and a couple of MetricReporters (which basically just duplicate the JMX and admin page reporting that's there at the moment, but which should be more extensible). The patch includes a change to RequestHandlerBase showing how this could work. The idea would be to eventually replace the getStatistics() call on SolrInfoMBean with this instead.

The next step would be to allow more MetricReporters to be defined in solrconfig.xml. The Metrics library comes with ganglia and graphite reporting modules, and we can add contrib plugins for both of those.

There's some more general cleanup that could be done around SolrInfoMBean (we've got two plugin handlers at /mbeans and /plugins that basically do the same thing, and the beans themselves have some weirdly inconsistent data on them - getVersion() returns different things for different impls, and getSource() seems pretty useless), but maybe that's for another issue.

Issue Links

• contains: SOLR-9731 Add jvm-wide JMX statistics for Solr
• is related to: SOLR-6586 JmxMonitoredMap#getAttribute is not very efficient.
• is related to: SOLR-6788 Expose Solr version via MBean
• is related to: SOLR-9462 Report aggregated cache stats for all cores on a Solr node
• is related to: SOLR-8785 Use Metrics library for core metrics
• is required by: SOLR-9854 Collect metrics for index merges and index store IO
• is required by: SOLR-9856 Collect metrics for shard replication and tlog replay on replicas
• is required by: SOLR-9857 Collect aggregated metrics from replicas in shard leader
• is required by: SOLR-9858 Collect aggregated metrics from nodes and shard leaders in overseer
• relates to: SOLR-5095 SolrCore.infoRegistry needs overhauled with some form of "namespacing"
• relates to: SOLR-15023 Timeout Issue with Solr Metrics API
• relates to: SOLR-9805 Use metrics-jvm library to instrument jvm internals
• relates to: SOLR-6757 SolrInfoMBean should be an abstract class rather than an interface.
• relates to: SOLR-9788 Use instrumented jetty classes
• links to: GitHub Pull Request #119
• links to: GitHub Pull Request #120
• links to: GitHub Pull Request #120