Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.4, 7.0
    • Component/s: metrics
    • Labels:
      None

      Description

      Following on from a discussion on the mailing list:
      http://search-lucene.com/m/IO0EI1qdyJF1/codahale&subj=Solr+metrics+in+Codahale+metrics+and+Graphite+

      It would be good to make Solr play more nicely with existing devops monitoring systems, such as Graphite or Ganglia. Stats monitoring at the moment is poll-only, either via JMX or through the admin stats page. I'd like to refactor things a bit to make this more pluggable.

      This patch is a start. It adds a new interface, InstrumentedBean, which extends SolrInfoMBean to return a [Metrics] MetricRegistry, and a couple of MetricReporters (which basically just duplicate the JMX and admin page reporting that's there at the moment, but which should be more extensible). The patch includes a change to RequestHandlerBase showing how this could work. The idea would be to eventually replace the getStatistics() call on SolrInfoMBean with this instead.

      The next step would be to allow more MetricReporters to be defined in solrconfig.xml. The Metrics library comes with ganglia and graphite reporting modules, and we can add contrib plugins for both of those.

      There's some more general cleanup that could be done around SolrInfoMBean (we've got two plugin handlers at /mbeans and /plugins that basically do the same thing, and the beans themselves have some weirdly inconsistent data on them - getVersion() returns different things for different impls, and getSource() seems pretty useless), but maybe that's for another issue.

      1. screenshot-2.png
        425 kB
        Andrzej Bialecki
      2. SOLR-4735.patch
        222 kB
        Andrzej Bialecki
      3. SOLR-4735.patch
        181 kB
        Andrzej Bialecki
      4. SOLR-4735.patch
        55 kB
        Christine Poerschke
      5. SOLR-4735.patch
        53 kB
        Alan Woodward
      6. SOLR-4735.patch
        52 kB
        Alan Woodward
      7. SOLR-4735.patch
        47 kB
        Alan Woodward

        Issue Links

          Activity

          Hide
          romseygeek Alan Woodward added a comment -

          Here's the patch.

          NB: this uses the same metrics library that I tried to use in my ill-starred attempts at SOLR-1972. However, the various leaky thread abstractions that were causing problems there have all been removed from this version.

          The JMX reporting doesn't quite work yet either, as it's dependent on being able to set the domains per-bean, and that functionality hasn't been released yet.

          Show
          romseygeek Alan Woodward added a comment - Here's the patch. NB: this uses the same metrics library that I tried to use in my ill-starred attempts at SOLR-1972 . However, the various leaky thread abstractions that were causing problems there have all been removed from this version. The JMX reporting doesn't quite work yet either, as it's dependent on being able to set the domains per-bean, and that functionality hasn't been released yet.
          Hide
          romseygeek Alan Woodward added a comment -

          New patch, moving everything to a single registry per-core, and adding a graphite reporter in contrib/.

          JMX naming still isn't working right, and it needs some tests, but I think this is a decent way forward. More eyes welcome...

          Show
          romseygeek Alan Woodward added a comment - New patch, moving everything to a single registry per-core, and adding a graphite reporter in contrib/. JMX naming still isn't working right, and it needs some tests, but I think this is a decent way forward. More eyes welcome...
          Hide
          ryantxu Ryan McKinley added a comment -

          This looks like it creates a new registry for every core (am I reading that wrong?) If so, I think sharing one registry would be best.

          Can the registry be in the CoreContainer rather then the core?

          I guess that would involve some cleanup when a core is unloaded, but it would let us share a single registry across cores and other apps (the case I am actually concerned with)

          Show
          ryantxu Ryan McKinley added a comment - This looks like it creates a new registry for every core (am I reading that wrong?) If so, I think sharing one registry would be best. Can the registry be in the CoreContainer rather then the core? I guess that would involve some cleanup when a core is unloaded, but it would let us share a single registry across cores and other apps (the case I am actually concerned with)
          Hide
          ryantxu Ryan McKinley added a comment -

          ideally CoreContainer could have a function like:

           
            MetricsRegistry createMetricsRegistry( ?? config ) {
              return new MetricsRegistry();
            }
          

          This would let other applications slip in their own registry – that already has reporting hooked up!

          Show
          ryantxu Ryan McKinley added a comment - ideally CoreContainer could have a function like: MetricsRegistry createMetricsRegistry( ?? config ) { return new MetricsRegistry(); } This would let other applications slip in their own registry – that already has reporting hooked up!
          Hide
          romseygeek Alan Woodward added a comment -

          That's a nice idea. The MetricsReporters would have to be able to build a filter that meant they only reported on Metrics set up for their core, but that should be doable. Will work on a patch over the weekend.

          Show
          romseygeek Alan Woodward added a comment - That's a nice idea. The MetricsReporters would have to be able to build a filter that meant they only reported on Metrics set up for their core, but that should be doable. Will work on a patch over the weekend.
          Hide
          romseygeek Alan Woodward added a comment -

          New patch. This upgrades to Metrics-3.0.0-BETA3, which adds the JMX domain-setting code, so JMX reporting now works correctly. It also implements Ryan's idea of moving the MetricRegistry to the CoreContainer.

          Show
          romseygeek Alan Woodward added a comment - New patch. This upgrades to Metrics-3.0.0-BETA3, which adds the JMX domain-setting code, so JMX reporting now works correctly. It also implements Ryan's idea of moving the MetricRegistry to the CoreContainer.
          Hide
          tim.potter Timothy Potter added a comment -

          I have some interest in working on this idea and wanted to get a sense where things sit / current thinking on this topic.

          Show
          tim.potter Timothy Potter added a comment - I have some interest in working on this idea and wanted to get a sense where things sit / current thinking on this topic.
          Hide
          romseygeek Alan Woodward added a comment -

          The patch that's here is pretty out-of-date, and seriously incomplete, but I still think it's a workable idea. I just don't have time to work on it... feel free to assign this one to yourself and have a go.

          Show
          romseygeek Alan Woodward added a comment - The patch that's here is pretty out-of-date, and seriously incomplete, but I still think it's a workable idea. I just don't have time to work on it... feel free to assign this one to yourself and have a go.
          Hide
          hossman Hoss Man added a comment -

          Tim: see also SOLR-5095 which points out some anoying problems we have with the lack of namespacing of plugins/mbeans right now that make it confusing/impossible to get stats from certain things.

          I think Alan's earlier point about how SolrInfoMBean cleanup is needed in general is spot on. We should keep an open mind to what the ideal APIs look like for monitoring/metrics and feel free to gut what we have in Solr5 to move towards those APIs

          Show
          hossman Hoss Man added a comment - Tim: see also SOLR-5095 which points out some anoying problems we have with the lack of namespacing of plugins/mbeans right now that make it confusing/impossible to get stats from certain things. I think Alan's earlier point about how SolrInfoMBean cleanup is needed in general is spot on. We should keep an open mind to what the ideal APIs look like for monitoring/metrics and feel free to gut what we have in Solr5 to move towards those APIs
          Hide
          markrmiller@gmail.com Mark Miller added a comment -

          feel free to gut what we have in Solr5

          We don't have a lot of time, but it would be great to solve SOLR-6586 - it really requires a different stats API to be sensible I think. It's a little tricky to make nice, but really the API calls for each individual attribute should be able to be calculated independently. Otherwise, there is just so much recalculation that it's hard to have everything be live and fast and even if you only want to fetch a single fast attribute, you will be penalized by the slowest.

          If you currently use a tool to enumerate and look at each attribute for monitoring, because of the duplicate bean issue and SOLR-6586, you can check the size of a directory like 40 times or something crazy when it really only had to be checked once. There is an API mismatch.

          Show
          markrmiller@gmail.com Mark Miller added a comment - feel free to gut what we have in Solr5 We don't have a lot of time, but it would be great to solve SOLR-6586 - it really requires a different stats API to be sensible I think. It's a little tricky to make nice, but really the API calls for each individual attribute should be able to be calculated independently. Otherwise, there is just so much recalculation that it's hard to have everything be live and fast and even if you only want to fetch a single fast attribute, you will be penalized by the slowest. If you currently use a tool to enumerate and look at each attribute for monitoring, because of the duplicate bean issue and SOLR-6586 , you can check the size of a directory like 40 times or something crazy when it really only had to be checked once. There is an API mismatch.
          Hide
          romseygeek Alan Woodward added a comment -

          I did some thinking on this over the weekend. Here's what I think the API should look like:

          • we use dropwizard metrics: https://dropwizard.github.io/metrics/3.1.0/. This is the same library I used in the initial patch, moved on a bit.
          • each core has its own MetricRegistry
          • SolrInfoMBean is replaced by an interface called SolrMetricsProducer, which has just three methods:
            • String getName()
            • Category getCategory()
            • void registerMetrics(MetricsCollector registry)
          • when core.inform() is called with the SolrMetricsProducer, it creates a MetricsCollector using the name and category, and then passes it to registerMetrics(). The producer can then add its own metrics as individual Counter, Gauge, Histogram, etc, instances to the collector. The Core then adds all these to its MetricRegistry, with appropriate names.
          • getStatistics() calls are replaced by a NamedListMetricReporter, as in the initial patch
          • the metrics JmxReporter will have to be extended slightly so that it knows about corehash values, to make sure that we deal with core reloads correctly.
          • CoreContainer could have its own MetricRegistry for any node-wide stats (things like System/Memory/JVM info - there's a metrics/jvm subproject which has a bunch of useful stats that we could add here).

          This way, all the individual stats are independent, and reporting and collection are nicely isolated.

          I also suggest we bin getSource(), getVersion() and getDocs() as they're generally left unimplemented, and probably getDescription() as well. And the Category enum could be replaced with a plain String, to allow plugins to add their own types.

          Show
          romseygeek Alan Woodward added a comment - I did some thinking on this over the weekend. Here's what I think the API should look like: we use dropwizard metrics: https://dropwizard.github.io/metrics/3.1.0/ . This is the same library I used in the initial patch, moved on a bit. each core has its own MetricRegistry SolrInfoMBean is replaced by an interface called SolrMetricsProducer, which has just three methods: String getName() Category getCategory() void registerMetrics(MetricsCollector registry) when core.inform() is called with the SolrMetricsProducer, it creates a MetricsCollector using the name and category, and then passes it to registerMetrics(). The producer can then add its own metrics as individual Counter, Gauge, Histogram, etc, instances to the collector. The Core then adds all these to its MetricRegistry, with appropriate names. getStatistics() calls are replaced by a NamedListMetricReporter, as in the initial patch the metrics JmxReporter will have to be extended slightly so that it knows about corehash values, to make sure that we deal with core reloads correctly. CoreContainer could have its own MetricRegistry for any node-wide stats (things like System/Memory/JVM info - there's a metrics/jvm subproject which has a bunch of useful stats that we could add here). This way, all the individual stats are independent, and reporting and collection are nicely isolated. I also suggest we bin getSource(), getVersion() and getDocs() as they're generally left unimplemented, and probably getDescription() as well. And the Category enum could be replaced with a plain String, to allow plugins to add their own types.
          Hide
          shalinmangar Shalin Shekhar Mangar added a comment - - edited

          That sounds like a reasonable start, Alan. Do you have any ideas on how we can introspect/discover metrics supported by a plugin? Also, does the metrics API have any support for merging stats from different servers? The last two questions are important to build aggregate stats API for SolrCloud (see SOLR-6325) and to easily build the next generation of Solr's Admin UI.

          We should also add stats support to the SearchComponents (maybe as a separate issue).

          Show
          shalinmangar Shalin Shekhar Mangar added a comment - - edited That sounds like a reasonable start, Alan. Do you have any ideas on how we can introspect/discover metrics supported by a plugin? Also, does the metrics API have any support for merging stats from different servers? The last two questions are important to build aggregate stats API for SolrCloud (see SOLR-6325 ) and to easily build the next generation of Solr's Admin UI. We should also add stats support to the SearchComponents (maybe as a separate issue).
          Hide
          romseygeek Alan Woodward added a comment -

          re introspection and discovery, each MetricProvider class will register metrics with a common prefix, and it should be easy to write a handler that returns all metrics with a given prefix.

          Merging stats isn't directly supported by the Metrics library, as far as I know.

          I'm halfway through a patch on this - LazyRequestHandlers are causing me a headache at the moment, but I think I can see a way through.

          Show
          romseygeek Alan Woodward added a comment - re introspection and discovery, each MetricProvider class will register metrics with a common prefix, and it should be easy to write a handler that returns all metrics with a given prefix. Merging stats isn't directly supported by the Metrics library, as far as I know. I'm halfway through a patch on this - LazyRequestHandlers are causing me a headache at the moment, but I think I can see a way through.
          Hide
          andyetitmoves Ramkumar Aiyengar added a comment -

          Alan Woodward, still quite interested in this, would you happen to have the current state in a patch and an idea of things which are still left?

          Show
          andyetitmoves Ramkumar Aiyengar added a comment - Alan Woodward , still quite interested in this, would you happen to have the current state in a patch and an idea of things which are still left?
          Hide
          wunder Walter Underwood added a comment -

          Anybody using the CodaHale metrics.jetty9.InstrumentedHandler? It looks a lot like something we built for our own use with Solr 4.

          http://metrics.dropwizard.io/3.1.0/manual/jetty/
          http://metrics.dropwizard.io/3.1.0/apidocs/com/codahale/metrics/jetty9/InstrumentedHandler.html

          wunder

          Show
          wunder Walter Underwood added a comment - Anybody using the CodaHale metrics.jetty9.InstrumentedHandler? It looks a lot like something we built for our own use with Solr 4. http://metrics.dropwizard.io/3.1.0/manual/jetty/ http://metrics.dropwizard.io/3.1.0/apidocs/com/codahale/metrics/jetty9/InstrumentedHandler.html wunder
          Hide
          jwartes Jeff Wartes added a comment -

          I have, and am, by instantiating a SharedMetricRegistry and GraphiteReporter directly in the jetty.xml. (Which is hacky, but in lieu of SOLR-8785, does work fine)
          I'm also using the logging and JVM metrics plugins quite happily.

          Show
          jwartes Jeff Wartes added a comment - I have, and am, by instantiating a SharedMetricRegistry and GraphiteReporter directly in the jetty.xml. (Which is hacky, but in lieu of SOLR-8785 , does work fine) I'm also using the logging and JVM metrics plugins quite happily.
          Hide
          cpoerschke Christine Poerschke added a comment -

          As I mentioned yesterday in SOLR-8785, in our team my colleague Kelvin Wong is currently also working on metrics stuff.

          The attached patch is Kelvin's work. (I only rebased it on top of the now committed SOLR-8785 changes and made minor adjustments in a couple of places.)

          Comments, questions, reviews, etc. welcome as usual. Thank you.


          SOLR-4735: pluggable SolrMetricReporter support e.g. SolrJmxReporter

          • SolrMetric(Info|Producer|Manager|Reporter) classes with tests
          • SolrJmxReporter class with test
          • SolrCore changes:
            • instantiates a SolrMetricManager
            • loads any configured SolrMetricReporters into the SolrMetricManager
            • registerInfoBean includes registering with the SolrMetricManager (for beans that implement the SolrMetricProducer interface)
          • RequestHandlerBase changes:
            • codahale metrics.Meter used instead of java atomic.LongAdder to count errors and timeouts
            • getMetrics() method (implementing SolrMetricProducer interface) added

          next planned steps:

          • couple of minor TODO's in the code
          • add test which uses a solrconfig.xml that configures (test) SolrMetricReporter(s)
          • 'ant beast' the new tests
          Show
          cpoerschke Christine Poerschke added a comment - As I mentioned yesterday in SOLR-8785 , in our team my colleague Kelvin Wong is currently also working on metrics stuff. The attached patch is Kelvin's work. (I only rebased it on top of the now committed SOLR-8785 changes and made minor adjustments in a couple of places.) Comments, questions, reviews, etc. welcome as usual. Thank you. SOLR-4735 : pluggable SolrMetricReporter support e.g. SolrJmxReporter SolrMetric(Info|Producer|Manager|Reporter) classes with tests SolrJmxReporter class with test SolrCore changes: instantiates a SolrMetricManager loads any configured SolrMetricReporters into the SolrMetricManager registerInfoBean includes registering with the SolrMetricManager (for beans that implement the SolrMetricProducer interface) RequestHandlerBase changes: codahale metrics.Meter used instead of java atomic.LongAdder to count errors and timeouts getMetrics() method (implementing SolrMetricProducer interface) added next planned steps: couple of minor TODO's in the code add test which uses a solrconfig.xml that configures (test) SolrMetricReporter(s) 'ant beast' the new tests
          Hide
          jwartes Jeff Wartes added a comment -

          For what it's worth, this looks like really great stuff to me.
          I'm still unconvinced that metrics should always get reset on core reload, which is a source of some complexity, but doing so is certainly consistent with the prior behavior, so I can hardly complain.
          I think I can see a path to providing reportable metrics outside of the RequestHandler. I'd be interested in Kelvin's thoughts on that subject though, since he chose not to use SharedMetricsRegistries.

          Show
          jwartes Jeff Wartes added a comment - For what it's worth, this looks like really great stuff to me. I'm still unconvinced that metrics should always get reset on core reload, which is a source of some complexity, but doing so is certainly consistent with the prior behavior, so I can hardly complain. I think I can see a path to providing reportable metrics outside of the RequestHandler. I'd be interested in Kelvin's thoughts on that subject though, since he chose not to use SharedMetricsRegistries.
          Hide
          kwong494 Kelvin Wong added a comment -

          Thanks Jeff.

          The attached patch really just piggybacks off of Alan's work and tries to flesh out the design. For now, only RequestHandlerBase exposes metrics through this framework. The idea is to eventually convert other SolrInfoMBeans into SolrMetricProducers so they can start providing reportable metrics too. This seems to be fairly doable.

          Re SharedMetricsRegistries: that's something we can definitely do. My rationale for not using it is so that logical groups of metrics can be nicely isolated at a per-core level. This ensures that any metric in a MetricManager's registry must have been registered through MetricManager::registerMetrics. A nice side-effect is that we can also store meta-information about each metric and pass that on to the reporters.

          I realize that using SharedMetricsRegistries provides a level of flexibility that this patch's approach does not. For example, if we wanted to share the registry on a CoreContainer level. I think there are ways around this and my personal preference is still for this logical grouping of metrics. But perhaps there may be use cases I'm neglecting to consider?

          Would be interested to hear your thoughts on this. Thanks!

          Show
          kwong494 Kelvin Wong added a comment - Thanks Jeff. The attached patch really just piggybacks off of Alan's work and tries to flesh out the design. For now, only RequestHandlerBase exposes metrics through this framework. The idea is to eventually convert other SolrInfoMBeans into SolrMetricProducers so they can start providing reportable metrics too. This seems to be fairly doable. Re SharedMetricsRegistries: that's something we can definitely do. My rationale for not using it is so that logical groups of metrics can be nicely isolated at a per-core level. This ensures that any metric in a MetricManager's registry must have been registered through MetricManager::registerMetrics. A nice side-effect is that we can also store meta-information about each metric and pass that on to the reporters. I realize that using SharedMetricsRegistries provides a level of flexibility that this patch's approach does not. For example, if we wanted to share the registry on a CoreContainer level. I think there are ways around this and my personal preference is still for this logical grouping of metrics. But perhaps there may be use cases I'm neglecting to consider? Would be interested to hear your thoughts on this. Thanks!
          Hide
          ab Andrzej Bialecki added a comment -

          I'm interested in getting this issue resolved, so I'd be happy to work on committing this (I asked Alan and he doesn't mind ).

          Christine Poerschke & Kelvin: great stuff, I really like the abstractions. I share Jeff's concern though that we need to consider how to maintain metrics that outlive any particular core instance (even core reload events ). Core reloads may be caused by several reasons (explicit action, config change, replication). I'm not sure under which scenario I'd prefer to reset metrics from a previous version of the core...

          Eventually we will want to instrument also other aspects of Solr, things that happen outside SolrCore (eg. SolrCloud operations, replication, leader metrics per replica, replica recovery stats, Jetty connections, heap, etc). For these using SharedMetricsRegistries would make more sense, so the question is whether we should use two different mechanisms for managing MetricRegistry instances, the other one being SolrMetricManager. Perhaps SolrMetricManager should use long-lived MetricRegistry instances that are managed in SharedMetricsRegistries?

          Also, from the point of view of monitoring the overall "load" of a particular node it would make sense to also track some really low-level Lucene stuff, such as major merges and read/write IO, but this can come later - let's first get the design right.

          Show
          ab Andrzej Bialecki added a comment - I'm interested in getting this issue resolved, so I'd be happy to work on committing this (I asked Alan and he doesn't mind ). Christine Poerschke & Kelvin: great stuff, I really like the abstractions. I share Jeff's concern though that we need to consider how to maintain metrics that outlive any particular core instance (even core reload events ). Core reloads may be caused by several reasons (explicit action, config change, replication). I'm not sure under which scenario I'd prefer to reset metrics from a previous version of the core... Eventually we will want to instrument also other aspects of Solr, things that happen outside SolrCore (eg. SolrCloud operations, replication, leader metrics per replica, replica recovery stats, Jetty connections, heap, etc). For these using SharedMetricsRegistries would make more sense, so the question is whether we should use two different mechanisms for managing MetricRegistry instances, the other one being SolrMetricManager . Perhaps SolrMetricManager should use long-lived MetricRegistry instances that are managed in SharedMetricsRegistries ? Also, from the point of view of monitoring the overall "load" of a particular node it would make sense to also track some really low-level Lucene stuff, such as major merges and read/write IO, but this can come later - let's first get the design right.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user sigram opened a pull request:

          https://github.com/apache/lucene-solr/pull/119

          SOLR-4735 Improve Solr metrics

          Branch created from the patch in Jira by Kelvin Wong.

          Changes include:

          • using shared instances of `MetricRegistry` per core.
          • unit test modifications.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/sigram/lucene-solr jira/solr-4735

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/lucene-solr/pull/119.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #119


          commit dba0663c79f7b27d4626152d36f8d6d4c62a878d
          Author: Andrzej Bialecki <andrzej.bialecki@lucidworks.com>
          Date: 2016-11-23T12:48:26Z

          Initial patch from Jira.

          commit 1ade9c443dbd5b9eae2ec5208b233d28fb20a8cb
          Author: Andrzej Bialecki <andrzej.bialecki@lucidworks.com>
          Date: 2016-11-24T10:45:20Z

          Merge branch 'master' into jira/solr-4735

          commit ba2a94fb52d21ed05053a098c8fb9919a469e5b3
          Author: Andrzej Bialecki <andrzej.bialecki@lucidworks.com>
          Date: 2016-11-24T15:32:04Z

          Use SharedMetricRegistries for managing per-core metrics.


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user sigram opened a pull request: https://github.com/apache/lucene-solr/pull/119 SOLR-4735 Improve Solr metrics Branch created from the patch in Jira by Kelvin Wong. Changes include: using shared instances of `MetricRegistry` per core. unit test modifications. You can merge this pull request into a Git repository by running: $ git pull https://github.com/sigram/lucene-solr jira/solr-4735 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/lucene-solr/pull/119.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #119 commit dba0663c79f7b27d4626152d36f8d6d4c62a878d Author: Andrzej Bialecki <andrzej.bialecki@lucidworks.com> Date: 2016-11-23T12:48:26Z Initial patch from Jira. commit 1ade9c443dbd5b9eae2ec5208b233d28fb20a8cb Author: Andrzej Bialecki <andrzej.bialecki@lucidworks.com> Date: 2016-11-24T10:45:20Z Merge branch 'master' into jira/solr-4735 commit ba2a94fb52d21ed05053a098c8fb9919a469e5b3 Author: Andrzej Bialecki <andrzej.bialecki@lucidworks.com> Date: 2016-11-24T15:32:04Z Use SharedMetricRegistries for managing per-core metrics.
          Hide
          shalinmangar Shalin Shekhar Mangar added a comment -

          I share Jeff's concern though that we need to consider how to maintain metrics that outlive any particular core instance (even core reload events )

          Having a reload reset stats is a bad idea. We can provide an explicit API to reset stats for a node or a core if required.

          Perhaps SolrMetricManager should use long-lived MetricRegistry instances that are managed in SharedMetricsRegistries?

          +1

          Also see the patch on SOLR-9788 which adds instrumented classes to Jetty that are managed by SharedMetricsRegistries. The JVM metrics can also be exposed in a similar way. It currently adds all jetty statistics to a metric registry named "solr" but we can split them out into multiple ones if needed. I'll rebase my patch over this pull request.

          As far as the pull request is concerned I'd suggest that we rename SolrMetricManager to SolrCoreMetricManager because it is tied to a single core.

          Show
          shalinmangar Shalin Shekhar Mangar added a comment - I share Jeff's concern though that we need to consider how to maintain metrics that outlive any particular core instance (even core reload events ) Having a reload reset stats is a bad idea. We can provide an explicit API to reset stats for a node or a core if required. Perhaps SolrMetricManager should use long-lived MetricRegistry instances that are managed in SharedMetricsRegistries? +1 Also see the patch on SOLR-9788 which adds instrumented classes to Jetty that are managed by SharedMetricsRegistries. The JVM metrics can also be exposed in a similar way. It currently adds all jetty statistics to a metric registry named "solr" but we can split them out into multiple ones if needed. I'll rebase my patch over this pull request. As far as the pull request is concerned I'd suggest that we rename SolrMetricManager to SolrCoreMetricManager because it is tied to a single core.
          Hide
          jwartes Jeff Wartes added a comment -

          From what I see, SolrMetricManager only needs the SolrCore for the config-based reporter instantiation, but that's a pretty nice thing to have.

          How about SolrMetricManager takes, as an optional second parameter to the constructor, the name of a SharedMetricRegistry. If absent, then it creates a new, isolated registry. With a name though, that means the config-based reporters you attach are actually being attached to the shared registry, pulling whatever happens to be in there too.
          Of course, then the core unregister action needs to be careful to only replace/reset those metrics that it'd added to the registry, instead of all of them as currently written. It could remove/replace the reporters with no real issue on every core reload (aside from possibly a blip in the reporting interval) though.

          Show
          jwartes Jeff Wartes added a comment - From what I see, SolrMetricManager only needs the SolrCore for the config-based reporter instantiation, but that's a pretty nice thing to have. How about SolrMetricManager takes, as an optional second parameter to the constructor, the name of a SharedMetricRegistry. If absent, then it creates a new, isolated registry. With a name though, that means the config-based reporters you attach are actually being attached to the shared registry, pulling whatever happens to be in there too. Of course, then the core unregister action needs to be careful to only replace/reset those metrics that it'd added to the registry, instead of all of them as currently written. It could remove/replace the reporters with no real issue on every core reload (aside from possibly a blip in the reporting interval) though.
          Hide
          kwong494 Kelvin Wong added a comment -

          Having a reload reset stats is a bad idea. We can provide an explicit API to reset stats for a node or a core if required.

          Right now, each producer creates/manages its own set of metrics. It might make sense to have some global object creating/managing all our metrics instead. Each producer can then call getOrCreate so that the same set of metrics are used across core reloads. My only concern here is how to namespace the metrics so that different producers clash on metric names. Perhaps give each producer access to the core name?

          Perhaps SolrMetricManager should use long-lived MetricRegistry instances that are managed in SharedMetricsRegistries?

          Sounds good. One suggestion I have is to namespace the registries. For example, we can have each SolrCore report on a registry, Jetty report on another, JVM on another, etc... Then we can configure reporters for each such registry by just specifying its name. This keeps the different sets of metrics nicely isolated and gives us flexibility as to how to report each set of metrics?

          Show
          kwong494 Kelvin Wong added a comment - Having a reload reset stats is a bad idea. We can provide an explicit API to reset stats for a node or a core if required. Right now, each producer creates/manages its own set of metrics. It might make sense to have some global object creating/managing all our metrics instead. Each producer can then call getOrCreate so that the same set of metrics are used across core reloads. My only concern here is how to namespace the metrics so that different producers clash on metric names. Perhaps give each producer access to the core name? Perhaps SolrMetricManager should use long-lived MetricRegistry instances that are managed in SharedMetricsRegistries? Sounds good. One suggestion I have is to namespace the registries. For example, we can have each SolrCore report on a registry, Jetty report on another, JVM on another, etc... Then we can configure reporters for each such registry by just specifying its name. This keeps the different sets of metrics nicely isolated and gives us flexibility as to how to report each set of metrics?
          Hide
          ab Andrzej Bialecki added a comment -

          Right now, each producer creates/manages its own set of metrics.

          Correct, but if the producer's life-cycle is tied to that of SolrCore then the lifecycle of that metric will be the same, ie it will vanish when that instance of core is closed.

          My only concern here is how to namespace the metrics so that different producers clash on metric names. Perhaps give each producer access to the core name?

          I think that the registration mechanism where SolrCore registers these metrics for each plugin is ok - it would be similar to how it works in your patch, except we would getOrCreate these metric instances instead of creating new ones for each registration.

          Sounds good. One suggestion I have is to namespace the registries.

          +1. In the pull request I used a core name, but that would be insufficient - it needs to be made into a hierarchy, eg. "/core/<coreName>" for SolrCore metrics, "/jetty/..." for Jetty metrics, etc.

          Show
          ab Andrzej Bialecki added a comment - Right now, each producer creates/manages its own set of metrics. Correct, but if the producer's life-cycle is tied to that of SolrCore then the lifecycle of that metric will be the same, ie it will vanish when that instance of core is closed. My only concern here is how to namespace the metrics so that different producers clash on metric names. Perhaps give each producer access to the core name? I think that the registration mechanism where SolrCore registers these metrics for each plugin is ok - it would be similar to how it works in your patch, except we would getOrCreate these metric instances instead of creating new ones for each registration. Sounds good. One suggestion I have is to namespace the registries. +1. In the pull request I used a core name, but that would be insufficient - it needs to be made into a hierarchy, eg. "/core/<coreName>" for SolrCore metrics, "/jetty/..." for Jetty metrics, etc.
          Hide
          jwartes Jeff Wartes added a comment -

          I had a scheme for collapsable namespaced registries in my original PR for SOLR-8785.

          Show
          jwartes Jeff Wartes added a comment - I had a scheme for collapsable namespaced registries in my original PR for SOLR-8785 .
          Hide
          shalinmangar Shalin Shekhar Mangar added a comment -

          I have created a new branch called "feature/metrics" for SOLR-9788, this and other future metrics enhancements – http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/497212e0

          Let's use this for integration between different patches.

          Show
          shalinmangar Shalin Shekhar Mangar added a comment - I have created a new branch called "feature/metrics" for SOLR-9788 , this and other future metrics enhancements – http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/497212e0 Let's use this for integration between different patches.
          Hide
          ab Andrzej Bialecki added a comment -

          Jeff Wartes thanks for the pointer to your PR, I borrowed parts of your code and updated my PR:

          • simplified and renamed SolrMetricManager to SolrCoreMetricManager as it really is specific to managing metrics related to SolrCore.
          • added a global component for registry management SolrMetricManager, which mostly offers useful syntactic sugar for working with SharedMetricRegistries

          Next step: I'm going to merge my work into Shalin Shekhar Mangar's branch.

          Show
          ab Andrzej Bialecki added a comment - Jeff Wartes thanks for the pointer to your PR, I borrowed parts of your code and updated my PR: simplified and renamed SolrMetricManager to SolrCoreMetricManager as it really is specific to managing metrics related to SolrCore . added a global component for registry management SolrMetricManager , which mostly offers useful syntactic sugar for working with SharedMetricRegistries Next step: I'm going to merge my work into Shalin Shekhar Mangar 's branch.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user sigram commented on the issue:

          https://github.com/apache/lucene-solr/pull/119

          This work will be merged and continued in the `features/metrics` branch at ASF.

          Show
          githubbot ASF GitHub Bot added a comment - Github user sigram commented on the issue: https://github.com/apache/lucene-solr/pull/119 This work will be merged and continued in the `features/metrics` branch at ASF.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user sigram closed the pull request at:

          https://github.com/apache/lucene-solr/pull/119

          Show
          githubbot ASF GitHub Bot added a comment - Github user sigram closed the pull request at: https://github.com/apache/lucene-solr/pull/119
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user sigram opened a pull request:

          https://github.com/apache/lucene-solr/pull/120

          SOLR-4735 Improve Solr metrics reporting

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/sigram/lucene-solr jira/solr-4735

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/lucene-solr/pull/120.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #120


          commit dba0663c79f7b27d4626152d36f8d6d4c62a878d
          Author: Andrzej Bialecki <andrzej.bialecki@lucidworks.com>
          Date: 2016-11-23T12:48:26Z

          Initial patch from Jira.

          commit 1ade9c443dbd5b9eae2ec5208b233d28fb20a8cb
          Author: Andrzej Bialecki <andrzej.bialecki@lucidworks.com>
          Date: 2016-11-24T10:45:20Z

          Merge branch 'master' into jira/solr-4735

          commit ba2a94fb52d21ed05053a098c8fb9919a469e5b3
          Author: Andrzej Bialecki <andrzej.bialecki@lucidworks.com>
          Date: 2016-11-24T15:32:04Z

          Use SharedMetricRegistries for managing per-core metrics.

          commit 7199e818da503bc0e8a40fed7d6a1f742ecbae55
          Author: Andrzej Bialecki <andrzej.bialecki@lucidworks.com>
          Date: 2016-11-24T15:44:55Z

          Revert accidental changes to this file.

          commit a4514638df8a2528f339107869b03fe568856fd9
          Author: Andrzej Bialecki <andrzej.bialecki@lucidworks.com>
          Date: 2016-11-28T14:01:42Z

          SOLR-4735 Use separate registries for core and other stuff. Use collapsible and
          configurable namespace.

          commit bf424d1ec1602dffeb33ab0acc8f470e351a6959
          Author: Kevin Risden <krisden@apache.org>
          Date: 2016-11-22T19:22:16Z

          SOLR-9728: Ability to specify Key Store type in solr.in file for SSL

          commit 5b2594350df11ef54d52f417b34c6d082ad85e89
          Author: Noble Paul <noble@apache.org>
          Date: 2016-11-29T02:35:47Z

          SOLR-9784: added deprecation javadocs

          commit 32c4bd7cc0ac2e93e833f5fe84be4ff69f0b7aeb
          Author: Noble Paul <noble@apache.org>
          Date: 2016-11-29T02:36:26Z

          Merge remote-tracking branch 'origin/master'

          commit 9b4f99c1b351b1401e2dd5922a06d79a809332fb
          Author: Andrzej Bialecki <andrzej.bialecki@lucidworks.com>
          Date: 2016-11-29T10:37:14Z

          SOLR-4735 Reorder args from top level to bottom.

          commit 70b358960dfe8a6da35991b2a84c93cc9370c3d8
          Author: Noble Paul <noble@apache.org>
          Date: 2016-11-29T12:32:59Z

          SOLR-9546: remove unnecessary boxing

          commit 2af8ec2689610f4dfb1f5d87b069b0eb54f72155
          Author: Andrzej Bialecki <andrzej.bialecki@lucidworks.com>
          Date: 2016-11-29T13:48:44Z

          SOLR-4735 More cleanup and generalization of JMX reporter.

          commit 4b7002eae75839d2f56a17a65d7e789cb71a9b26
          Author: Andrzej Bialecki <andrzej.bialecki@lucidworks.com>
          Date: 2016-11-29T13:56:24Z

          Merge branch 'master' into jira/solr-4735


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user sigram opened a pull request: https://github.com/apache/lucene-solr/pull/120 SOLR-4735 Improve Solr metrics reporting You can merge this pull request into a Git repository by running: $ git pull https://github.com/sigram/lucene-solr jira/solr-4735 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/lucene-solr/pull/120.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #120 commit dba0663c79f7b27d4626152d36f8d6d4c62a878d Author: Andrzej Bialecki <andrzej.bialecki@lucidworks.com> Date: 2016-11-23T12:48:26Z Initial patch from Jira. commit 1ade9c443dbd5b9eae2ec5208b233d28fb20a8cb Author: Andrzej Bialecki <andrzej.bialecki@lucidworks.com> Date: 2016-11-24T10:45:20Z Merge branch 'master' into jira/solr-4735 commit ba2a94fb52d21ed05053a098c8fb9919a469e5b3 Author: Andrzej Bialecki <andrzej.bialecki@lucidworks.com> Date: 2016-11-24T15:32:04Z Use SharedMetricRegistries for managing per-core metrics. commit 7199e818da503bc0e8a40fed7d6a1f742ecbae55 Author: Andrzej Bialecki <andrzej.bialecki@lucidworks.com> Date: 2016-11-24T15:44:55Z Revert accidental changes to this file. commit a4514638df8a2528f339107869b03fe568856fd9 Author: Andrzej Bialecki <andrzej.bialecki@lucidworks.com> Date: 2016-11-28T14:01:42Z SOLR-4735 Use separate registries for core and other stuff. Use collapsible and configurable namespace. commit bf424d1ec1602dffeb33ab0acc8f470e351a6959 Author: Kevin Risden <krisden@apache.org> Date: 2016-11-22T19:22:16Z SOLR-9728 : Ability to specify Key Store type in solr.in file for SSL commit 5b2594350df11ef54d52f417b34c6d082ad85e89 Author: Noble Paul <noble@apache.org> Date: 2016-11-29T02:35:47Z SOLR-9784 : added deprecation javadocs commit 32c4bd7cc0ac2e93e833f5fe84be4ff69f0b7aeb Author: Noble Paul <noble@apache.org> Date: 2016-11-29T02:36:26Z Merge remote-tracking branch 'origin/master' commit 9b4f99c1b351b1401e2dd5922a06d79a809332fb Author: Andrzej Bialecki <andrzej.bialecki@lucidworks.com> Date: 2016-11-29T10:37:14Z SOLR-4735 Reorder args from top level to bottom. commit 70b358960dfe8a6da35991b2a84c93cc9370c3d8 Author: Noble Paul <noble@apache.org> Date: 2016-11-29T12:32:59Z SOLR-9546 : remove unnecessary boxing commit 2af8ec2689610f4dfb1f5d87b069b0eb54f72155 Author: Andrzej Bialecki <andrzej.bialecki@lucidworks.com> Date: 2016-11-29T13:48:44Z SOLR-4735 More cleanup and generalization of JMX reporter. commit 4b7002eae75839d2f56a17a65d7e789cb71a9b26 Author: Andrzej Bialecki <andrzej.bialecki@lucidworks.com> Date: 2016-11-29T13:56:24Z Merge branch 'master' into jira/solr-4735
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit db1339ff622cc73871897f8b345a9be19134a95e in lucene-solr's branch refs/heads/feature/metrics from Andrzej Bialecki
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=db1339f ]

          SOLR-4735 Improve Solr metric reporting (Alan Woodward, Kelvin Wong,
          Christine Poerschke, Jeff Wartes, ab)

          Show
          jira-bot ASF subversion and git services added a comment - Commit db1339ff622cc73871897f8b345a9be19134a95e in lucene-solr's branch refs/heads/feature/metrics from Andrzej Bialecki [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=db1339f ] SOLR-4735 Improve Solr metric reporting (Alan Woodward, Kelvin Wong, Christine Poerschke, Jeff Wartes, ab)
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit f027098b4c0a27c70c7cb33dc80492a199bc44cf in lucene-solr's branch refs/heads/feature/metrics from Andrzej Bialecki
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=f027098 ]

          SOLR-4735 Cleanup and fix lint errors.

          Show
          jira-bot ASF subversion and git services added a comment - Commit f027098b4c0a27c70c7cb33dc80492a199bc44cf in lucene-solr's branch refs/heads/feature/metrics from Andrzej Bialecki [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=f027098 ] SOLR-4735 Cleanup and fix lint errors.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user randomstatistic commented on a diff in the pull request:

          https://github.com/apache/lucene-solr/pull/120#discussion_r90144072

          — Diff: solr/core/src/java/org/apache/solr/metrics/SolrMetricManager.java —
          @@ -0,0 +1,216 @@
          +/*
          + * Licensed to the Apache Software Foundation (ASF) under one or more
          + * contributor license agreements. See the NOTICE file distributed with
          + * this work for additional information regarding copyright ownership.
          + * The ASF licenses this file to You under the Apache License, Version 2.0
          + * (the "License"); you may not use this file except in compliance with
          + * the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +package org.apache.solr.metrics;
          +
          +import java.util.Set;
          +
          +import com.codahale.metrics.Counter;
          +import com.codahale.metrics.Histogram;
          +import com.codahale.metrics.Meter;
          +import com.codahale.metrics.MetricFilter;
          +import com.codahale.metrics.MetricRegistry;
          +import com.codahale.metrics.SharedMetricRegistries;
          +import com.codahale.metrics.Snapshot;
          +import com.codahale.metrics.Timer;
          +import org.apache.solr.common.util.NamedList;
          +
          +/**
          + *
          + */
          +public class SolrMetricManager {
          +
          + public static final String REGISTRY_NAME_PREFIX = "solr";
          + public static final String DEFAULT_REGISTRY = MetricRegistry.name(REGISTRY_NAME_PREFIX, "default");
          +
          + // don't create instances of this class
          + private SolrMetricManager() { }
          +
          +
          + /**
          + * Return a set of existing registry names.
          + */
          + public static Set<String> registryNames()

          { + return SharedMetricRegistries.names(); + }

          +
          + /**
          + * Get (or create if not present) a named registry
          + * @param registry name of the registry
          + * @return existing or newly created registry
          + */
          + public static MetricRegistry registryFor(String registry)

          { + return SharedMetricRegistries.getOrCreate(overridableRegistryName(registry)); + }

          +
          + /**
          + * Remove all metrics from a specified registry.
          + * @param registry registry name
          + */
          + public static void clearRegistryFor(String registry) {
          + SharedMetricRegistries.getOrCreate(overridableRegistryName(registry)).removeMatching(MetricFilter.ALL);
          — End diff –

          This, and several other places below could delegate to `registryFor(registry)`

          Show
          githubbot ASF GitHub Bot added a comment - Github user randomstatistic commented on a diff in the pull request: https://github.com/apache/lucene-solr/pull/120#discussion_r90144072 — Diff: solr/core/src/java/org/apache/solr/metrics/SolrMetricManager.java — @@ -0,0 +1,216 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.solr.metrics; + +import java.util.Set; + +import com.codahale.metrics.Counter; +import com.codahale.metrics.Histogram; +import com.codahale.metrics.Meter; +import com.codahale.metrics.MetricFilter; +import com.codahale.metrics.MetricRegistry; +import com.codahale.metrics.SharedMetricRegistries; +import com.codahale.metrics.Snapshot; +import com.codahale.metrics.Timer; +import org.apache.solr.common.util.NamedList; + +/** + * + */ +public class SolrMetricManager { + + public static final String REGISTRY_NAME_PREFIX = "solr"; + public static final String DEFAULT_REGISTRY = MetricRegistry.name(REGISTRY_NAME_PREFIX, "default"); + + // don't create instances of this class + private SolrMetricManager() { } + + + /** + * Return a set of existing registry names. + */ + public static Set<String> registryNames() { + return SharedMetricRegistries.names(); + } + + /** + * Get (or create if not present) a named registry + * @param registry name of the registry + * @return existing or newly created registry + */ + public static MetricRegistry registryFor(String registry) { + return SharedMetricRegistries.getOrCreate(overridableRegistryName(registry)); + } + + /** + * Remove all metrics from a specified registry. + * @param registry registry name + */ + public static void clearRegistryFor(String registry) { + SharedMetricRegistries.getOrCreate(overridableRegistryName(registry)).removeMatching(MetricFilter.ALL); — End diff – This, and several other places below could delegate to `registryFor(registry)`
          Hide
          jwartes Jeff Wartes added a comment -

          Heh, I wondered whether something like that would happen if I commented on github. Should I constrain myself to talking in Jira?

          Show
          jwartes Jeff Wartes added a comment - Heh, I wondered whether something like that would happen if I commented on github. Should I constrain myself to talking in Jira?
          Hide
          ab Andrzej Bialecki added a comment -

          Ouch! Yeah, let's keep the discussion here. Unfortunately, the way Github integration is set up now it makes pull requests painful to work with.

          So... originally I wasn't sure how often that method would be called and I wanted to save the cost of an additional method call - but it's called so infrequently that elegance indeed seems more important.

          Also, I merged the latest code from the PR into branch feature/metrics because we need to coordinate this work with Shalin Shekhar Mangar, so I'll continue working there. I'll update the PR anyway for completeness.

          Show
          ab Andrzej Bialecki added a comment - Ouch! Yeah, let's keep the discussion here. Unfortunately, the way Github integration is set up now it makes pull requests painful to work with. So... originally I wasn't sure how often that method would be called and I wanted to save the cost of an additional method call - but it's called so infrequently that elegance indeed seems more important. Also, I merged the latest code from the PR into branch feature/metrics because we need to coordinate this work with Shalin Shekhar Mangar , so I'll continue working there. I'll update the PR anyway for completeness.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 7cedc4acb26c23d11218037bfcad0737c3c566b6 in lucene-solr's branch refs/heads/feature/metrics from Andrzej Bialecki
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=7cedc4a ]

          SOLR-4735 Simplify the code by using registryFor(String).

          Show
          jira-bot ASF subversion and git services added a comment - Commit 7cedc4acb26c23d11218037bfcad0737c3c566b6 in lucene-solr's branch refs/heads/feature/metrics from Andrzej Bialecki [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=7cedc4a ] SOLR-4735 Simplify the code by using registryFor(String).
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 14ff79175f524b38ec85f0542753e28dc8b1b2c6 in lucene-solr's branch refs/heads/feature/metrics from Andrzej Bialecki
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=14ff791 ]

          SOLR-4735 Simplify method names. Allow removing multiple metrics with a prefix.

          Show
          jira-bot ASF subversion and git services added a comment - Commit 14ff79175f524b38ec85f0542753e28dc8b1b2c6 in lucene-solr's branch refs/heads/feature/metrics from Andrzej Bialecki [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=14ff791 ] SOLR-4735 Simplify method names. Allow removing multiple metrics with a prefix.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 38887c4a446990b8c37195c08d9bb63108dad31e in lucene-solr's branch refs/heads/feature/metrics from Andrzej Bialecki
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=38887c4 ]

          SOLR-4735 Clean up the API. Remove metrics on core unload.

          Show
          jira-bot ASF subversion and git services added a comment - Commit 38887c4a446990b8c37195c08d9bb63108dad31e in lucene-solr's branch refs/heads/feature/metrics from Andrzej Bialecki [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=38887c4 ] SOLR-4735 Clean up the API. Remove metrics on core unload.
          Hide
          shalinmangar Shalin Shekhar Mangar added a comment -

          I built Solr from the feature/metric branch and tried it out. I have a few questions/comments:

          1. I expected that the SolrJmxReporter would be enabled by default but it is not. Should it be? Eventually we should get rid of our current JMX integration (maybe in 7.0?) so it makes sense to have the alternative enabled by default.
          2. If or how does the SolrJmxReporter work with the <jmx> tag in solrconfig.xml? Does that get deprecated eventually?
          3. There is no test solrconfig.xml which has a reporter section in it. There should be at least one with the jmx reporter configured that we test instead of just relying on code to create a new metric manager and add a reporter to it.
          4. The example solrconfig.xml should have a sample reporter section even if it is commented out if the default jmx reporter is not enabled by default.
          5. The metric reporter should be configurable via the Config API
          6. Do we want to support Graphite or Ganglia reporters as well?

          The last two can be worked upon in separate issues.

          Show
          shalinmangar Shalin Shekhar Mangar added a comment - I built Solr from the feature/metric branch and tried it out. I have a few questions/comments: I expected that the SolrJmxReporter would be enabled by default but it is not. Should it be? Eventually we should get rid of our current JMX integration (maybe in 7.0?) so it makes sense to have the alternative enabled by default. If or how does the SolrJmxReporter work with the <jmx> tag in solrconfig.xml? Does that get deprecated eventually? There is no test solrconfig.xml which has a reporter section in it. There should be at least one with the jmx reporter configured that we test instead of just relying on code to create a new metric manager and add a reporter to it. The example solrconfig.xml should have a sample reporter section even if it is commented out if the default jmx reporter is not enabled by default. The metric reporter should be configurable via the Config API Do we want to support Graphite or Ganglia reporters as well? The last two can be worked upon in separate issues.
          Hide
          ab Andrzej Bialecki added a comment -

          Ad 1. Indeed, there is some inconsistency here. I think we should deprecate the <jmx>, and turn on SolrJmxReporter by default in cloud mode (in the example non-cloud config <jmx> is turned off).
          Ad 2. currently it's independent. Yes, I think we should eventually remove the <jmx> section.
          Ad 3. Good point, I'll create one.
          Ad 4. Right, I'll add this.
          Ad 5. Not sure how to do that, let's discuss this offline.
          Ad 6. This would be easy to add but it brings several additional dependencies from metrics-graphite and metrics-ganglia artifacts. Are we ok with that?

          Show
          ab Andrzej Bialecki added a comment - Ad 1. Indeed, there is some inconsistency here. I think we should deprecate the <jmx> , and turn on SolrJmxReporter by default in cloud mode (in the example non-cloud config <jmx> is turned off). Ad 2. currently it's independent. Yes, I think we should eventually remove the <jmx> section. Ad 3. Good point, I'll create one. Ad 4. Right, I'll add this. Ad 5. Not sure how to do that, let's discuss this offline. Ad 6. This would be easy to add but it brings several additional dependencies from metrics-graphite and metrics-ganglia artifacts. Are we ok with that?
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit fea0e200a8083ebd86d8e522939e4977d072bbe7 in lucene-solr's branch refs/heads/feature/metrics from Kelvin Wong
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=fea0e20 ]

          SOLR-4735: SolrMetricsIntegrationTest (Kelvin Wong via Christine Poerschke)

          Adds SolrMetricsIntegrationTest which uses solrconfig-metricreporter.xml which configures MockMetricReporter instances.

          also:

          • JmxUtil and SolrJmxReporter tweaks
          • SolrMetricReporterTest.MockReporter turned into MockMetricReporter
          • changes in SolrCoreMetricManagerTest and SolrJmxReporterTest:
            • moved initCore from BeforeClass to Before(Test) so that After(Test) can do deleteCore
            • TODO: verify interaction between tests (SolrCoreMetricManagerTest and SolrMetricsIntegrationTest and SolrJmxReporterTest)
          • SolrCoreMetricManagerTest instead of SolrJmxReporter use MockMetricReporter
          Show
          jira-bot ASF subversion and git services added a comment - Commit fea0e200a8083ebd86d8e522939e4977d072bbe7 in lucene-solr's branch refs/heads/feature/metrics from Kelvin Wong [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=fea0e20 ] SOLR-4735 : SolrMetricsIntegrationTest (Kelvin Wong via Christine Poerschke) Adds SolrMetricsIntegrationTest which uses solrconfig-metricreporter.xml which configures MockMetricReporter instances. also: JmxUtil and SolrJmxReporter tweaks SolrMetricReporterTest.MockReporter turned into MockMetricReporter changes in SolrCoreMetricManagerTest and SolrJmxReporterTest: moved initCore from BeforeClass to Before(Test) so that After(Test) can do deleteCore TODO: verify interaction between tests (SolrCoreMetricManagerTest and SolrMetricsIntegrationTest and SolrJmxReporterTest) SolrCoreMetricManagerTest instead of SolrJmxReporter use MockMetricReporter
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit f489bb8566985174111d4e91df2d6ec03ffcb01e in lucene-solr's branch refs/heads/feature/metrics from Andrzej Bialecki
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=f489bb8 ]

          SOLR-4735 This method may actually remove several metrics.

          Show
          jira-bot ASF subversion and git services added a comment - Commit f489bb8566985174111d4e91df2d6ec03ffcb01e in lucene-solr's branch refs/heads/feature/metrics from Andrzej Bialecki [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=f489bb8 ] SOLR-4735 This method may actually remove several metrics.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 1bc01d25ef2fa8fa015ba2fdb5b2ffc8ae3cec0a in lucene-solr's branch refs/heads/feature/metrics from Andrzej Bialecki
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=1bc01d2 ]

          SOLR-4735 Introduce a concept of top-level group of metrics for things
          that happen outside SolrCore.

          Show
          jira-bot ASF subversion and git services added a comment - Commit 1bc01d25ef2fa8fa015ba2fdb5b2ffc8ae3cec0a in lucene-solr's branch refs/heads/feature/metrics from Andrzej Bialecki [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=1bc01d2 ] SOLR-4735 Introduce a concept of top-level group of metrics for things that happen outside SolrCore.
          Hide
          ab Andrzej Bialecki added a comment -

          Thanks Kelvin for creating the new tests! You beat me to it...

          After some discussion with Shalin it looks like we need to be able to manage SolrMetricReporter-s outside the SolrCore context - after all they need to also report metrics that are not related to SolrCore:

          • I added a notion of "group" of metrics, which corresponds to a top-level subsystem in a Solr node (I'm not attached to the name, naming is hard, so if you have a better suggestion please speak up):
              enum Group { jvm, jetty, http, node, core }
            

            These correspond to JVM, Jetty, HttpClient, SolrCore, and node-level metrics such as Zookeeper stats, Overseer queues etc. The idea is that if you fire JConsole you should see something like this:

            * solr
              - jvm
              - jetty
              - http
              - node
                - zookeeper
                - overseer
                ...
              - core
                - collection1
                  - CORE
                  - QUERYHANDLER
                  …
            
          • Since we need reporters for these top-level groups we need to be able to configure them outside SolrCore config. On the other hand we probably don't want to always specify the same reporters in each collection config, too. I'll try to refactor the code so that we can specify the base reporter configs in CoreContainer configuration, and then override them per collection if needed.
          • Current implementation instantiates separate reporters for each collection. I'll look into reusing single global-level reporters when possible, and creating new instances only if there are per-collection overrides.
          • Currently we use separate instances of MetricRegistry per core. This may seem wasteful but if you consider deployments with thousands of cores, and multiply this by hundreds of metrics per core then any lookups in a HashMap with 100k keys, or filtering by prefix, will cause performance issues. So I think the current design is more scalable than using a single registry for all cores.
          Show
          ab Andrzej Bialecki added a comment - Thanks Kelvin for creating the new tests! You beat me to it... After some discussion with Shalin it looks like we need to be able to manage SolrMetricReporter -s outside the SolrCore context - after all they need to also report metrics that are not related to SolrCore : I added a notion of "group" of metrics, which corresponds to a top-level subsystem in a Solr node (I'm not attached to the name, naming is hard, so if you have a better suggestion please speak up): enum Group { jvm, jetty, http, node, core } These correspond to JVM, Jetty, HttpClient, SolrCore, and node-level metrics such as Zookeeper stats, Overseer queues etc. The idea is that if you fire JConsole you should see something like this: * solr - jvm - jetty - http - node - zookeeper - overseer ... - core - collection1 - CORE - QUERYHANDLER … Since we need reporters for these top-level groups we need to be able to configure them outside SolrCore config. On the other hand we probably don't want to always specify the same reporters in each collection config, too. I'll try to refactor the code so that we can specify the base reporter configs in CoreContainer configuration, and then override them per collection if needed. Current implementation instantiates separate reporters for each collection. I'll look into reusing single global-level reporters when possible, and creating new instances only if there are per-collection overrides. Currently we use separate instances of MetricRegistry per core. This may seem wasteful but if you consider deployments with thousands of cores, and multiply this by hundreds of metrics per core then any lookups in a HashMap with 100k keys, or filtering by prefix, will cause performance issues. So I think the current design is more scalable than using a single registry for all cores.
          Hide
          kwong494 Kelvin Wong added a comment -

          Hi Andrzej,

          I added a notion of "group" of metrics, which corresponds to a top-level subsystem in a Solr node

          • Nice! I really like this concept. Will we be instantiating separate reporters for each `Group` then? That way, reporting can be more flexibly configured. (ex. Jetty goes to JMX and Graphite, JVM goes to only JMX, etc...)

          I'll look into reusing single global-level reporters when possible, and creating new instances only if there are per-collection overrides.

          • It looks like JmxReporter takes only one `MetricRegistry` at a time (and GraphiteReporter, etc. for that matter). Will we need to build some sort of `AggregateMetricRegistry` to join each core's registries? Or do you have something else in mind?
          • On a separate note, it would be nice if we could just specify which registry we'd like a reporter to attach to. So for example, we can attach one reporter to `collection1`, another to `zookeeper`, and one more to `jvm`. These are at different levels in the metrics hierarchy but perhaps we can just pass in the registry's name as part of the config for a reporter?
          Show
          kwong494 Kelvin Wong added a comment - Hi Andrzej, I added a notion of "group" of metrics, which corresponds to a top-level subsystem in a Solr node Nice! I really like this concept. Will we be instantiating separate reporters for each `Group` then? That way, reporting can be more flexibly configured. (ex. Jetty goes to JMX and Graphite, JVM goes to only JMX, etc...) I'll look into reusing single global-level reporters when possible, and creating new instances only if there are per-collection overrides. It looks like JmxReporter takes only one `MetricRegistry` at a time (and GraphiteReporter , etc. for that matter). Will we need to build some sort of `AggregateMetricRegistry` to join each core's registries? Or do you have something else in mind? On a separate note, it would be nice if we could just specify which registry we'd like a reporter to attach to. So for example, we can attach one reporter to `collection1`, another to `zookeeper`, and one more to `jvm`. These are at different levels in the metrics hierarchy but perhaps we can just pass in the registry's name as part of the config for a reporter?
          Hide
          jwartes Jeff Wartes added a comment -

          `MetricRegistry` is really just a bunch of convenience methods and thread-safety around a `MetricSet`. There isn't much overhead difference between the two. But really, when I think of a MetricRegistry, I think of it as "a set of metrics I want to attach a reporter to", nothing more.
          It's a bit disappointing that reporters take a Registry instead of a MetricSet, since a Registry isa MetricSet.

          With that in mind, one strategy would be have every logical grouping of metrics use its own dedicated (probably shared) registry, and then bind the reporter-registry concept together at reporter definition time.

          That is, create a non-shared registry explicitly for the purpose of attaching a reporter to it, and only when asked to define a reporter. The reporter definition would then include the names of the registries to be reported. Under the hood, a new registry would be created as the union of the requested registries, and the reporter instantiated and attached to that. We'd have to make sure the namespace of all the metrics in the metric groups is unique, so that arbitrary groups can be combined without conflict, but that sounds desirable regardless.

          Show
          jwartes Jeff Wartes added a comment - `MetricRegistry` is really just a bunch of convenience methods and thread-safety around a `MetricSet`. There isn't much overhead difference between the two. But really, when I think of a MetricRegistry, I think of it as "a set of metrics I want to attach a reporter to", nothing more. It's a bit disappointing that reporters take a Registry instead of a MetricSet, since a Registry isa MetricSet. With that in mind, one strategy would be have every logical grouping of metrics use its own dedicated (probably shared) registry, and then bind the reporter-registry concept together at reporter definition time. That is, create a non-shared registry explicitly for the purpose of attaching a reporter to it, and only when asked to define a reporter. The reporter definition would then include the names of the registries to be reported. Under the hood, a new registry would be created as the union of the requested registries, and the reporter instantiated and attached to that. We'd have to make sure the namespace of all the metrics in the metric groups is unique, so that arbitrary groups can be combined without conflict, but that sounds desirable regardless.
          Hide
          ab Andrzej Bialecki added a comment - - edited

          Will we be instantiating separate reporters for each `Group` then?

          Part of the refactoring I'm working on is moving reporter configs to solr.xml under <solr><metricReporters><reporter name="graphite" group="jvm" class="..."/> ... . Then appropriate reporters would be created for each group at the time when the component that manages this group of metrics is initialized (eg. "core" on SolrCore creation, "node" when CoreContainer is loaded etc).

          Regarding reporters that could take multiple registries ... yeah, it seems a waste to create separate reporters for each core if they have identical configs. I'm not sure yet how to solve this - eg. for JMX reporting any sort of aggregate reporter would have to create multiple JMXReporter-s anyway, one per registry, because that's how the API is implemented.

          it would be nice if we could just specify which registry we'd like a reporter to attach to

          Hmm, we could perhaps use either group or registry attribute in the reporter config.
          (edit: ugh, Markdown vs Jira markup)

          Show
          ab Andrzej Bialecki added a comment - - edited Will we be instantiating separate reporters for each `Group` then? Part of the refactoring I'm working on is moving reporter configs to solr.xml under <solr><metricReporters><reporter name="graphite" group="jvm" class="..."/> ... . Then appropriate reporters would be created for each group at the time when the component that manages this group of metrics is initialized (eg. "core" on SolrCore creation, "node" when CoreContainer is loaded etc). Regarding reporters that could take multiple registries ... yeah, it seems a waste to create separate reporters for each core if they have identical configs. I'm not sure yet how to solve this - eg. for JMX reporting any sort of aggregate reporter would have to create multiple JMXReporter -s anyway, one per registry, because that's how the API is implemented. it would be nice if we could just specify which registry we'd like a reporter to attach to Hmm, we could perhaps use either group or registry attribute in the reporter config. (edit: ugh, Markdown vs Jira markup)
          Hide
          ab Andrzej Bialecki added a comment -

          `MetricRegistry` is really just a bunch of convenience methods and thread-safety around a `MetricSet`

          Well, my comment referred to the fact that MetricRegistry actually uses ConcurrentHashMap for keeping metrics, and having a map with 100k+ keys is never good. But I agree the API could have been more flexible - if reporters were taking MetricSet we could fake one either from multiple registries or from a subset of metrics from one registry, or a combination thereof.

          We can implement an aggregating franken-registry by overriding all methods in MetricRegistry to always delegate operations to sub-registries. It's a little bit hackish but doable. We could create these as non-shared registries only for the purpose of reporting.

          Show
          ab Andrzej Bialecki added a comment - `MetricRegistry` is really just a bunch of convenience methods and thread-safety around a `MetricSet` Well, my comment referred to the fact that MetricRegistry actually uses ConcurrentHashMap for keeping metrics, and having a map with 100k+ keys is never good. But I agree the API could have been more flexible - if reporters were taking MetricSet we could fake one either from multiple registries or from a subset of metrics from one registry, or a combination thereof. We can implement an aggregating franken-registry by overriding all methods in MetricRegistry to always delegate operations to sub-registries. It's a little bit hackish but doable. We could create these as non-shared registries only for the purpose of reporting.
          Hide
          jwartes Jeff Wartes added a comment -

          Yeah, I get that. I like this line of thought because it means we can create as many registries as make sense, (cores, collections, logical code sections, etc) without worrying about how to get everything reported. We only have to pick some names.

          What about a class that extends MetricRegistry and also implements MetricRegistryListener? Call that a ListeningMetricRegistry or something. When the configuration asks for a reporter on some set of (registry) names, we create a new, perhaps non-shared ListeningMetricRegistry, use registerAll to scoop the metrics in the desired registries into it, and then call addListener on all the desired registries with the ListeningMetricRegistry so everything stays in sync?

          So that could still mean a single registry with a ton of metrics, but only in cases where there's been an explicit request for a reporter on a ton of metrics.

          Show
          jwartes Jeff Wartes added a comment - Yeah, I get that. I like this line of thought because it means we can create as many registries as make sense, (cores, collections, logical code sections, etc) without worrying about how to get everything reported. We only have to pick some names. What about a class that extends MetricRegistry and also implements MetricRegistryListener? Call that a ListeningMetricRegistry or something. When the configuration asks for a reporter on some set of (registry) names, we create a new, perhaps non-shared ListeningMetricRegistry, use registerAll to scoop the metrics in the desired registries into it, and then call addListener on all the desired registries with the ListeningMetricRegistry so everything stays in sync? So that could still mean a single registry with a ton of metrics, but only in cases where there's been an explicit request for a reporter on a ton of metrics.
          Hide
          kwong494 Kelvin Wong added a comment -

          Hmm wouldn't these aggregate registries defeat the point of keeping them separate in the first place (from a performance perspective)? For example, if a user configures a JMXReporter and a GraphiteReporter on all registries, Solr would have to basically make two copies of all of its registries.

          Perhaps we can just "fake" an aggregate reporter? There can be configuration logic so that one reporter is instantiated for each registry that the user configured. This might be a bit wasteful but we won't have to deal with maintaining an aggregate registry or writing reporters that do the aggregation. And to the user, it seems as though they only needed to configure one reporter.

          Show
          kwong494 Kelvin Wong added a comment - Hmm wouldn't these aggregate registries defeat the point of keeping them separate in the first place (from a performance perspective)? For example, if a user configures a JMXReporter and a GraphiteReporter on all registries, Solr would have to basically make two copies of all of its registries. Perhaps we can just "fake" an aggregate reporter? There can be configuration logic so that one reporter is instantiated for each registry that the user configured. This might be a bit wasteful but we won't have to deal with maintaining an aggregate registry or writing reporters that do the aggregation. And to the user, it seems as though they only needed to configure one reporter.
          Hide
          jwartes Jeff Wartes added a comment -

          That seems pretty viable too. As I mentioned, the memory overhead of a registry is pretty low, just a concurrent map and a list. Plus, the actual metric objects in the map would be shared by both registries, so I'd be more concerned about the work involved keeping them synchronized then with just having multiple registries.

          I confess though, I don't have a clear idea whether that's more or less overhead than multiple identically-configured reporters. It feels like most of the possible performance issues here are linear, so it may not matter. Two reporters iterating through 10 metrics each sounds pretty much the same as one reporter iterating over 20 to me, all else being equal.

          Show
          jwartes Jeff Wartes added a comment - That seems pretty viable too. As I mentioned, the memory overhead of a registry is pretty low, just a concurrent map and a list. Plus, the actual metric objects in the map would be shared by both registries, so I'd be more concerned about the work involved keeping them synchronized then with just having multiple registries. I confess though, I don't have a clear idea whether that's more or less overhead than multiple identically-configured reporters. It feels like most of the possible performance issues here are linear, so it may not matter. Two reporters iterating through 10 metrics each sounds pretty much the same as one reporter iterating over 20 to me, all else being equal.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 46c662fcab56906f3fa6fde09d3789d1d2fc6aed in lucene-solr's branch refs/heads/feature/metrics from Andrzej Bialecki
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=46c662f ]

          SOLR-4735 WIP: move metric reporter config to CoreContainer level. Manage
          reporters in SolrMetricManager.

          Show
          jira-bot ASF subversion and git services added a comment - Commit 46c662fcab56906f3fa6fde09d3789d1d2fc6aed in lucene-solr's branch refs/heads/feature/metrics from Andrzej Bialecki [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=46c662f ] SOLR-4735 WIP: move metric reporter config to CoreContainer level. Manage reporters in SolrMetricManager.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 6f14f41044f6ec5b58c3328fd7186e3f8e1a9d33 in lucene-solr's branch refs/heads/feature/metrics from Andrzej Bialecki
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=6f14f41 ]

          SOLR-4735 Move metric reporter configuration to solr.xml:

          • add metric reporting at "node" level,
          • introduce reporters applicable to multiple groups, multiple registries or all groups,
            based on the value of "group" and "registry" attributes,
          • handle core rename and swap operations so that metrics are preserved but renamed.
          • modify SolrJmxReporter to split the hierarchy based also on the reporter's name.
          Show
          jira-bot ASF subversion and git services added a comment - Commit 6f14f41044f6ec5b58c3328fd7186e3f8e1a9d33 in lucene-solr's branch refs/heads/feature/metrics from Andrzej Bialecki [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=6f14f41 ] SOLR-4735 Move metric reporter configuration to solr.xml: add metric reporting at "node" level, introduce reporters applicable to multiple groups, multiple registries or all groups, based on the value of "group" and "registry" attributes, handle core rename and swap operations so that metrics are preserved but renamed. modify SolrJmxReporter to split the hierarchy based also on the reporter's name.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit cff7097f40488bb97fb8c38a76ad28d2e1ae50d2 in lucene-solr's branch refs/heads/feature/metrics from Andrzej Bialecki
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=cff7097 ]

          SOLR-4735 Fix SolrMetricsIntegrationTest. Fix registration of "solr.jvm" metrics.
          Add a few helper methods to SolrMetricManager.

          Show
          jira-bot ASF subversion and git services added a comment - Commit cff7097f40488bb97fb8c38a76ad28d2e1ae50d2 in lucene-solr's branch refs/heads/feature/metrics from Andrzej Bialecki [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=cff7097 ] SOLR-4735 Fix SolrMetricsIntegrationTest. Fix registration of "solr.jvm" metrics. Add a few helper methods to SolrMetricManager.
          Hide
          ab Andrzej Bialecki added a comment - - edited

          Brief summary of the current state on the branch:

          • SolrMetricManager is now the single central location responsible for managing instances of MetricRegistry and SolrMetricReporter-s.
          • plugin configurations are declared in solr.xml:
            <solr>
              <metrics>
                <reporter name="foo" group="node, core" class="org.apache.solr.metrics.reporters.SolrJmxReporter"/>
              ...
              </metrics>
            ...
            </solr>
            
          • each reporter plugin can be configured so that it's automatically instantiated:
            • for a specific "group" or "registry". These attributes may contain multiple comma- or space-separated values, in which case the reporter will be instantiated for each matching group / registry.
            • for every registry, when neither "group" or "registry" attribute is specified.
          • reporters are instantiated on component initialization and closed when a component is closed / shut down. Currently this happens in: CoreContainer for node group, SolrCore for core group and in SolrDispatchFilter for jvm group.
          • instances of metrics are maintained across core reloads and renames (or swaps). They are only permanently deleted on core delete.
          • SolrJmxReporter now maintains a hierarchy of ObjectName-s that consists of the following:
            • registry name (split on dots into sub-paths)
            • reporter name
            • category
            • scope
            • metric type
            • metric name

          See the attached JConsole screenshot.

          To do, possibly in another issues:

          • add metrics for "jetty" and "http" groups.
          • add a handler to report all this wealth of information
          • add other reporter implementations - SLF4J reporter is a low-hanging fruit, and it doesn't bring any additional dependencies.

          Comments, suggestions, review and patches are welcome!

          Show
          ab Andrzej Bialecki added a comment - - edited Brief summary of the current state on the branch: SolrMetricManager is now the single central location responsible for managing instances of MetricRegistry and SolrMetricReporter -s. plugin configurations are declared in solr.xml : <solr> <metrics> <reporter name= "foo" group= "node, core" class= "org.apache.solr.metrics.reporters.SolrJmxReporter" /> ... </metrics> ... </solr> each reporter plugin can be configured so that it's automatically instantiated: for a specific "group" or "registry". These attributes may contain multiple comma- or space-separated values, in which case the reporter will be instantiated for each matching group / registry. for every registry, when neither "group" or "registry" attribute is specified. reporters are instantiated on component initialization and closed when a component is closed / shut down. Currently this happens in: CoreContainer for node group, SolrCore for core group and in SolrDispatchFilter for jvm group. instances of metrics are maintained across core reloads and renames (or swaps). They are only permanently deleted on core delete. SolrJmxReporter now maintains a hierarchy of ObjectName -s that consists of the following: registry name (split on dots into sub-paths) reporter name category scope metric type metric name See the attached JConsole screenshot. To do, possibly in another issues: add metrics for "jetty" and "http" groups. add a handler to report all this wealth of information add other reporter implementations - SLF4J reporter is a low-hanging fruit, and it doesn't bring any additional dependencies. Comments, suggestions, review and patches are welcome!
          Hide
          andyetitmoves Ramkumar Aiyengar added a comment -

          Didn't Shalin already add a /admin/metrics endpoint for reporting metrics?

          Show
          andyetitmoves Ramkumar Aiyengar added a comment - Didn't Shalin already add a /admin/metrics endpoint for reporting metrics?
          Hide
          shalinmangar Shalin Shekhar Mangar added a comment -

          Didn't Shalin already add a /admin/metrics endpoint for reporting metrics?

          In SOLR-9812, I've added the metrics servlet supplied by the library but it is not longer useful for the flexible registry scheme implemented here. I'm writing a custom handler for it now. I will upload a patch soon.

          Show
          shalinmangar Shalin Shekhar Mangar added a comment - Didn't Shalin already add a /admin/metrics endpoint for reporting metrics? In SOLR-9812 , I've added the metrics servlet supplied by the library but it is not longer useful for the flexible registry scheme implemented here. I'm writing a custom handler for it now. I will upload a patch soon.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit ab52041c9bfea8285446b79f39ddfbf02eebc845 in lucene-solr's branch refs/heads/feature/metrics from Andrzej Bialecki
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ab52041 ]

          SOLR-4735 Test also core rename. Fix expectations when registries are reused across tests in
          the same JVM.

          Show
          jira-bot ASF subversion and git services added a comment - Commit ab52041c9bfea8285446b79f39ddfbf02eebc845 in lucene-solr's branch refs/heads/feature/metrics from Andrzej Bialecki [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ab52041 ] SOLR-4735 Test also core rename. Fix expectations when registries are reused across tests in the same JVM.
          Hide
          ab Andrzej Bialecki added a comment - - edited

          There is one important issue that is still unsolved in the current patch - collecting per-collection metrics at the same time as we collect per-core metrics. A single instance of Solr can hold multiple shards and / or replicas that belong to the same logical collection and it would be useful to get combined metrics at a collection level.

          If they weren't needed at the same time, we could probably set up per-core aliases (using overridableRegistryName or a similar mechanism) so that a single per-collection registry would be used for all shards/replicas. However, since all shards would then modify the same underlying Metric objects it would not be possible anymore to separate per-shard metrics from these aggregates. This is however the simple fallback solution of the problem - allow either per-core or per-collection metrics and never both.

          Codahale API doesn't support aggregation of child metrics - if it were possible then we could fake the aggregated metrics on the fly when they are needed.

          So far the only solution that comes to my mind that allows us to keep both levels of reporting is to extend Metric subclasses so that they delegate their updates to the parent instance(s), something like the following:

          public class ChildCounter extends Counter {
           public ChildCounter(Counter... parents) { ... }
          
           public void inc(long n) {
             super.inc(n);
             for (Counter c : parents) {
               c.inc(n);
             }
           }
          }
          

          I.e. all updates to the child instances would be applied at (nearly) the same time to parent instances - and parent instances will be referenced by several child instances from different shards. For example, the ChildCounter instance would be registered in "solr.core.collection1_shard1" registry, and the aggregate counter would be registered in "solr.core.collection1" registry, and the same aggregate counter would be used by ChildCounter from "solr.core.collection1_shard2".

          In order to maintain this delegation Solr components would have to always use SolrMetricRegsitry.counter(...), .timer(...), .meter(...), .histogram(...) methods that would set up this delegation, by obtaining metric instances from the parent registries and returning eg. ChildCounter instead of a regular Counter, ChildTimer instead of regular Timer etc.

          It's not clear however how to handle in this design shard / replica updates, eg. split or merge shards, add / remove replica, etc. For performance reasons the Child* metric implementations would use direct references to parent metrics, without having to do a lookup each time the list of parents is modified - which of course means that the list of parents will get out of sync pretty quickly in presence of cluster changes...

          Any other suggestions?

          Show
          ab Andrzej Bialecki added a comment - - edited There is one important issue that is still unsolved in the current patch - collecting per-collection metrics at the same time as we collect per-core metrics. A single instance of Solr can hold multiple shards and / or replicas that belong to the same logical collection and it would be useful to get combined metrics at a collection level. If they weren't needed at the same time, we could probably set up per-core aliases (using overridableRegistryName or a similar mechanism) so that a single per-collection registry would be used for all shards/replicas. However, since all shards would then modify the same underlying Metric objects it would not be possible anymore to separate per-shard metrics from these aggregates. This is however the simple fallback solution of the problem - allow either per-core or per-collection metrics and never both. Codahale API doesn't support aggregation of child metrics - if it were possible then we could fake the aggregated metrics on the fly when they are needed. So far the only solution that comes to my mind that allows us to keep both levels of reporting is to extend Metric subclasses so that they delegate their updates to the parent instance(s), something like the following: public class ChildCounter extends Counter { public ChildCounter(Counter... parents) { ... } public void inc( long n) { super .inc(n); for (Counter c : parents) { c.inc(n); } } } I.e. all updates to the child instances would be applied at (nearly) the same time to parent instances - and parent instances will be referenced by several child instances from different shards. For example, the ChildCounter instance would be registered in "solr.core.collection1_shard1" registry, and the aggregate counter would be registered in "solr.core.collection1" registry, and the same aggregate counter would be used by ChildCounter from "solr.core.collection1_shard2". In order to maintain this delegation Solr components would have to always use SolrMetricRegsitry.counter(...), .timer(...), .meter(...), .histogram(...) methods that would set up this delegation, by obtaining metric instances from the parent registries and returning eg. ChildCounter instead of a regular Counter , ChildTimer instead of regular Timer etc. It's not clear however how to handle in this design shard / replica updates, eg. split or merge shards, add / remove replica, etc. For performance reasons the Child* metric implementations would use direct references to parent metrics, without having to do a lookup each time the list of parents is modified - which of course means that the list of parents will get out of sync pretty quickly in presence of cluster changes... Any other suggestions?
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit c2291001356e6fa1bb72e0543c3e035b5f320862 in lucene-solr's branch refs/heads/feature/metrics from Andrzej Bialecki
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=c229100 ]

          SOLR-4735 WIP. Initial design for propagating updates to linked registries.

          Show
          jira-bot ASF subversion and git services added a comment - Commit c2291001356e6fa1bb72e0543c3e035b5f320862 in lucene-solr's branch refs/heads/feature/metrics from Andrzej Bialecki [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=c229100 ] SOLR-4735 WIP. Initial design for propagating updates to linked registries.
          Hide
          ab Andrzej Bialecki added a comment -

          After discussing this with Shalin I came to conclusion that core-level metrics should be sufficient, and the cost of this added complexity is too high, so I'm going to revert that last commit.

          Collection-level metrics wouldn't be complete anyway in general case - shards and their replicas would live on different nodes anyway, so the collection-level metrics from a single node would never represent the whole picture for a collection, just a partial one for the parts of a collection that live on a single node.

          Show
          ab Andrzej Bialecki added a comment - After discussing this with Shalin I came to conclusion that core-level metrics should be sufficient, and the cost of this added complexity is too high, so I'm going to revert that last commit. Collection-level metrics wouldn't be complete anyway in general case - shards and their replicas would live on different nodes anyway, so the collection-level metrics from a single node would never represent the whole picture for a collection, just a partial one for the parts of a collection that live on a single node.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 99799cafe57cc32fa2a6bf6c25de3acd18cea248 in lucene-solr's branch refs/heads/feature/metrics from Andrzej Bialecki
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=99799ca ]

          Revert "SOLR-4735 WIP. Initial design for propagating updates to linked registries."

          This reverts commit c2291001356e6fa1bb72e0543c3e035b5f320862.

          Show
          jira-bot ASF subversion and git services added a comment - Commit 99799cafe57cc32fa2a6bf6c25de3acd18cea248 in lucene-solr's branch refs/heads/feature/metrics from Andrzej Bialecki [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=99799ca ] Revert " SOLR-4735 WIP. Initial design for propagating updates to linked registries." This reverts commit c2291001356e6fa1bb72e0543c3e035b5f320862.
          Hide
          jwartes Jeff Wartes added a comment - - edited

          I've fallen behind keeping up with your changes, but for what it's worth, I agree with this. Collection-level metrics are at the cluster level, in aggregate. It's up to the thing you're reporting the metrics into to do the aggregation. For example, what I really want on my dashboard in grafana is a line, something like:

          AVG(solr.[all nodes].[all cores belonging to a particular collection].latency.p95)

          Then I can drill into a particular node, or core, in my reporting tool if I want. There's a requirement that the metrics namespaces being reported allows for aggregation like this, which might mean a core needs to know the collection to which it belongs, but I don't think the node itself should needs to report collection metrics.

          Show
          jwartes Jeff Wartes added a comment - - edited I've fallen behind keeping up with your changes, but for what it's worth, I agree with this. Collection-level metrics are at the cluster level, in aggregate. It's up to the thing you're reporting the metrics into to do the aggregation. For example, what I really want on my dashboard in grafana is a line, something like: AVG(solr. [all nodes] . [all cores belonging to a particular collection] .latency.p95) Then I can drill into a particular node, or core, in my reporting tool if I want. There's a requirement that the metrics namespaces being reported allows for aggregation like this, which might mean a core needs to know the collection to which it belongs, but I don't think the node itself should needs to report collection metrics.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit d1227af722ef95494efc6c977a1c29f94108b4bf in lucene-solr's branch refs/heads/feature/metrics from Andrzej Bialecki
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=d1227af ]

          SOLR-4735 Javadoc plus test improvements.

          Show
          jira-bot ASF subversion and git services added a comment - Commit d1227af722ef95494efc6c977a1c29f94108b4bf in lucene-solr's branch refs/heads/feature/metrics from Andrzej Bialecki [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=d1227af ] SOLR-4735 Javadoc plus test improvements.
          Hide
          ab Andrzej Bialecki added a comment -

          Metric names already contain collection name in their namespace - eg. in SolrCloud a core name would be collection1_shard1_replica1. The full metrics name would be then something like solr.core.collection1_shard1_replica1.myReporter.QUERYHANDLER./admin/ping.counters.requests.Count, which seems awfully long to a human but is well-structured and easy to process in code.

          Show
          ab Andrzej Bialecki added a comment - Metric names already contain collection name in their namespace - eg. in SolrCloud a core name would be collection1_shard1_replica1 . The full metrics name would be then something like solr.core.collection1_shard1_replica1.myReporter.QUERYHANDLER./admin/ping.counters.requests.Count , which seems awfully long to a human but is well-structured and easy to process in code.
          Hide
          jwartes Jeff Wartes added a comment -

          Oh, one thing just occurred to me though. There are essentially two classes of request to a collection - the top-level request, and the per-shard fan-out requests. I guess you can sort of derive the metrics of the top-level request from the per-core metrics, but it requires you know the number of shards, and still only works if the two classes of request are not mixed together.

          Show
          jwartes Jeff Wartes added a comment - Oh, one thing just occurred to me though. There are essentially two classes of request to a collection - the top-level request, and the per-shard fan-out requests. I guess you can sort of derive the metrics of the top-level request from the per-core metrics, but it requires you know the number of shards, and still only works if the two classes of request are not mixed together.
          Hide
          jwartes Jeff Wartes added a comment -

          That's almost perfect. Can we replace those underscores with dots? That would mean the dashboard doesn't need to regex the "name" in order to group similar metrics.

          Show
          jwartes Jeff Wartes added a comment - That's almost perfect. Can we replace those underscores with dots? That would mean the dashboard doesn't need to regex the "name" in order to group similar metrics.
          Hide
          ab Andrzej Bialecki added a comment -

          Hmm, not blindly, you could have a collection named my_collection - but we could make check the collection name and split replica names into proper hierarchy, eg. my_collection_shard1_replica1 -> my_collection.shard1.replica1

          Show
          ab Andrzej Bialecki added a comment - Hmm, not blindly, you could have a collection named my_collection - but we could make check the collection name and split replica names into proper hierarchy, eg. my_collection_shard1_replica1 -> my_collection.shard1.replica1
          Hide
          jwartes Jeff Wartes added a comment -

          Understood, and not all cores are part of a collection. But if it matches the solrcloud convention, it would be pretty nice to use it. (and the node name if it doesn't) I could've sworn I saw an existing function for picking a node name apart somewhere, but I can't seem to find it now - maybe it was in a patch I read or something.

          Show
          jwartes Jeff Wartes added a comment - Understood, and not all cores are part of a collection. But if it matches the solrcloud convention, it would be pretty nice to use it. (and the node name if it doesn't) I could've sworn I saw an existing function for picking a node name apart somewhere, but I can't seem to find it now - maybe it was in a patch I read or something.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 98a04b6cca32d7b2a3da8b30291e1df1edd885cd in lucene-solr's branch refs/heads/feature/metrics from Andrzej Bialecki
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=98a04b6 ]

          SOLR-4735 Add unit test for SolrMetricManager. Other changes:

          • add metrics to SolrCore related to newSearcher events.
          • build a dot-separated hierarchical registry name for cores that are a part of
            SolrCloud collection.
          • don't split metrics by class in SolrJmxReporter - this only obscured the hierarchy.
          Show
          jira-bot ASF subversion and git services added a comment - Commit 98a04b6cca32d7b2a3da8b30291e1df1edd885cd in lucene-solr's branch refs/heads/feature/metrics from Andrzej Bialecki [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=98a04b6 ] SOLR-4735 Add unit test for SolrMetricManager. Other changes: add metrics to SolrCore related to newSearcher events. build a dot-separated hierarchical registry name for cores that are a part of SolrCloud collection. don't split metrics by class in SolrJmxReporter - this only obscured the hierarchy.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 3ba5d7db0971eb638a5e1dae17fb3c9ea36440d4 in lucene-solr's branch refs/heads/feature/metrics from Andrzej Bialecki
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=3ba5d7d ]

          SOLR-4735 Fix javadoc errors.

          Show
          jira-bot ASF subversion and git services added a comment - Commit 3ba5d7db0971eb638a5e1dae17fb3c9ea36440d4 in lucene-solr's branch refs/heads/feature/metrics from Andrzej Bialecki [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=3ba5d7d ] SOLR-4735 Fix javadoc errors.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 5f0637cc8569768ac9ce2a38cef5405163a974c0 in lucene-solr's branch refs/heads/feature/metrics from Shalin Shekhar Mangar
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=5f0637c ]

          SOLR-4735: Use method in java.util.Objects instead of the forbidden methods in Guava's Preconditions class

          Show
          jira-bot ASF subversion and git services added a comment - Commit 5f0637cc8569768ac9ce2a38cef5405163a974c0 in lucene-solr's branch refs/heads/feature/metrics from Shalin Shekhar Mangar [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=5f0637c ] SOLR-4735 : Use method in java.util.Objects instead of the forbidden methods in Guava's Preconditions class
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 92ef10dbdea6d8764d2a1ecba3d53abee542542d in lucene-solr's branch refs/heads/feature/metrics from Andrzej Bialecki
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=92ef10d ]

          SOLR-4735 Add gauges that report the current number of cores.

          Show
          jira-bot ASF subversion and git services added a comment - Commit 92ef10dbdea6d8764d2a1ecba3d53abee542542d in lucene-solr's branch refs/heads/feature/metrics from Andrzej Bialecki [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=92ef10d ] SOLR-4735 Add gauges that report the current number of cores.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit f56da1df6e92da5f1ab524caf62d30ea3a3ceede in lucene-solr's branch refs/heads/feature/metrics from Andrzej Bialecki
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=f56da1d ]

          SOLR-4735 Update CHANGES.txt

          Show
          jira-bot ASF subversion and git services added a comment - Commit f56da1df6e92da5f1ab524caf62d30ea3a3ceede in lucene-solr's branch refs/heads/feature/metrics from Andrzej Bialecki [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=f56da1d ] SOLR-4735 Update CHANGES.txt
          Hide
          dsmiley David Smiley added a comment -

          In the future, please create branches named SOLR-XXXX (or LUCENE-XXXX) which is the pattern expected by ASF's git JIRA notifier. Since this "metrics" feature-branch does not follow this convention, merge commits here trigger a bunch of notifications to other issues that were merged in.

          Alternatively if this is going to keep occurring... then maybe we can ask ASF Infra to modify the regexp that has this rule.

          Show
          dsmiley David Smiley added a comment - In the future, please create branches named SOLR-XXXX (or LUCENE-XXXX) which is the pattern expected by ASF's git JIRA notifier. Since this "metrics" feature-branch does not follow this convention, merge commits here trigger a bunch of notifications to other issues that were merged in. Alternatively if this is going to keep occurring... then maybe we can ask ASF Infra to modify the regexp that has this rule.
          Hide
          ab Andrzej Bialecki added a comment -

          Wilco. Sorry about that, I was wondering why this is so chatty...

          Show
          ab Andrzej Bialecki added a comment - Wilco. Sorry about that, I was wondering why this is so chatty...
          Hide
          shalinmangar Shalin Shekhar Mangar added a comment - - edited

          David Smiley - I didn't know that commits to jira/ branches are skipped by the commit bot before I created that branch. I figured that since multiple jira issues will be committed to this branch before we merge it to master, I should use a more descriptive name such as "feature/metrics". I have opened a ticket with ASF INFRA to skip notifications for commits on all branches beginning with feature/ for Lucene/Solr project. See https://issues.apache.org/jira/servicedesk/customer/portal/1/INFRA-13133

          Show
          shalinmangar Shalin Shekhar Mangar added a comment - - edited David Smiley - I didn't know that commits to jira/ branches are skipped by the commit bot before I created that branch. I figured that since multiple jira issues will be committed to this branch before we merge it to master, I should use a more descriptive name such as "feature/metrics". I have opened a ticket with ASF INFRA to skip notifications for commits on all branches beginning with feature/ for Lucene/Solr project. See https://issues.apache.org/jira/servicedesk/customer/portal/1/INFRA-13133
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 8bbdb6248c5de3f3bd61501ba42a50aeec29c78b in lucene-solr's branch refs/heads/master from Andrzej Bialecki
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=8bbdb62 ]

          Squashed commit of branch 'feature/metrics', containing:
          SOLR-4735: Improve Solr metrics reporting
          SOLR-9812: Implement /admin/metrics API
          SOLR-9805: Use metrics-jvm library to instrument jvm internals
          SOLR-9788: Use instrumented jetty classes

          Show
          jira-bot ASF subversion and git services added a comment - Commit 8bbdb6248c5de3f3bd61501ba42a50aeec29c78b in lucene-solr's branch refs/heads/master from Andrzej Bialecki [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=8bbdb62 ] Squashed commit of branch 'feature/metrics', containing: SOLR-4735 : Improve Solr metrics reporting SOLR-9812 : Implement /admin/metrics API SOLR-9805 : Use metrics-jvm library to instrument jvm internals SOLR-9788 : Use instrumented jetty classes
          Hide
          ab Andrzej Bialecki added a comment -

          Uploaded final diff between the branch and master, for completeness.

          Show
          ab Andrzej Bialecki added a comment - Uploaded final diff between the branch and master, for completeness.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit b37a72d941dba705b43c8868584bd9df775a1537 in lucene-solr's branch refs/heads/master from Andrzej Bialecki
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=b37a72d ]

          SOLR-4735 Use overridableRegistryName also for predefined shared registries.
          Cleanup + javadocs.

          Show
          jira-bot ASF subversion and git services added a comment - Commit b37a72d941dba705b43c8868584bd9df775a1537 in lucene-solr's branch refs/heads/master from Andrzej Bialecki [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=b37a72d ] SOLR-4735 Use overridableRegistryName also for predefined shared registries. Cleanup + javadocs.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 9dde8a30303de4bce5da45189219dd6150252b29 in lucene-solr's branch refs/heads/branch_6x from Andrzej Bialecki
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=9dde8a3 ]

          Cumulative patch from master, originally developed on branch
          'feature/metrics', which brings the following issues:

          • SOLR-4735: Improve Solr metrics reporting
          • SOLR-9812: Implement /admin/metrics API
          • SOLR-9805: Use metrics-jvm library to instrument jvm internals
          • SOLR-9788: Use instrumented jetty classes
          Show
          jira-bot ASF subversion and git services added a comment - Commit 9dde8a30303de4bce5da45189219dd6150252b29 in lucene-solr's branch refs/heads/branch_6x from Andrzej Bialecki [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=9dde8a3 ] Cumulative patch from master, originally developed on branch 'feature/metrics', which brings the following issues: SOLR-4735 : Improve Solr metrics reporting SOLR-9812 : Implement /admin/metrics API SOLR-9805 : Use metrics-jvm library to instrument jvm internals SOLR-9788 : Use instrumented jetty classes

            People

            • Assignee:
              ab Andrzej Bialecki
              Reporter:
              romseygeek Alan Woodward
            • Votes:
              8 Vote for this issue
              Watchers:
              28 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development