Solr
  1. Solr
  2. SOLR-1972

Need additional query stats in admin interface - median, 95th and 99th percentile

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 1.4
    • Fix Version/s: 4.1, 5.0
    • Component/s: web gui
    • Labels:
      None

      Description

      I would like to see more detailed query statistics from the admin GUI. This is what you can get now:

      requests : 809
      errors : 0
      timeouts : 0
      totalTime : 70053
      avgTimePerRequest : 86.59209
      avgRequestsPerSecond : 0.8148785

      I'd like to see more data on the time per request - median, 95th percentile, 99th percentile, and any other statistical function that makes sense to include. In my environment, the first bunch of queries after startup tend to take several seconds each. I find that the average value tends to be useless until it has several thousand queries under its belt and the caches are thoroughly warmed. The statistical functions I have mentioned would quickly eliminate the influence of those initial slow queries.

      The system will have to store individual data about each query. I don't know if this is something Solr does already. It would be nice to have a configurable count of how many of the most recent data points are kept, to control the amount of memory the feature uses. The default value could be something like 1024 or 4096.

      1. elyograg-1972-3.2.patch
        10 kB
        Shawn Heisey
      2. elyograg-1972-3.2.patch
        10 kB
        Shawn Heisey
      3. elyograg-1972-trunk.patch
        10 kB
        Shawn Heisey
      4. elyograg-1972-trunk.patch
        10 kB
        Shawn Heisey
      5. elyograg-closeable.patch
        37 kB
        Shawn Heisey
      6. leak.patch
        3 kB
        Robert Muir
      7. leak-closeable.patch
        13 kB
        Shawn Heisey
      8. revert-SOLR-1972.patch
        9 kB
        Shawn Heisey
      9. SOLR-1972_forked-metrics.patch
        53 kB
        Alan Woodward
      10. SOLR-1972_forked-metrics.patch
        57 kB
        Alan Woodward
      11. SOLR-1972_metrics.patch
        8 kB
        Shawn Heisey
      12. SOLR-1972_metrics.patch
        8 kB
        Alan Woodward
      13. SOLR-1972_metrics.patch
        8 kB
        Alan Woodward
      14. SOLR-1972_metrics.patch
        8 kB
        Alan Woodward
      15. SOLR-1972_metrics.patch
        8 kB
        Alan Woodward
      16. SOLR-1972_metrics.patch
        6 kB
        Shawn Heisey
      17. SOLR-1972_metrics.patch
        6 kB
        Alan Woodward
      18. SOLR-1972_metrics.patch
        5 kB
        Alan Woodward
      19. SOLR-1972.patch
        9 kB
        Adrien Grand
      20. SOLR-1972.patch
        9 kB
        Adrien Grand
      21. SOLR-1972.patch
        7 kB
        Adrien Grand
      22. SOLR-1972.patch
        8 kB
        Adrien Grand
      23. SOLR-1972-branch3x-url_pattern.patch
        11 kB
        Shawn Heisey
      24. SOLR-1972-branch4x.patch
        13 kB
        Shawn Heisey
      25. SOLR-1972-branch4x.patch
        13 kB
        Shawn Heisey
      26. solr1972-metricsregistry-branch4x-failure.log
        2.47 MB
        Shawn Heisey
      27. SOLR-1972-url_pattern.patch
        11 kB
        Eric Pugh
      28. stacktraces.tar.gz
        14 kB
        Shawn Heisey

        Issue Links

          Activity

          Hide
          Adrien Grand added a comment -

          This patches adds rolling statistics (same as non-rolling statistics, plus median and 75/90/99 percentiles) to request handlers.

          Rolling statistics are off by default because they require synchronization, but the critical section is very small so no overhead should be noticeable.

          Show
          Adrien Grand added a comment - This patches adds rolling statistics (same as non-rolling statistics, plus median and 75/90/99 percentiles) to request handlers. Rolling statistics are off by default because they require synchronization, but the critical section is very small so no overhead should be noticeable.
          Hide
          Shawn Heisey added a comment -

          Thank you, Adrien! I tried to apply this to Solr 3.2, which I am in the process of rolling into production, but it wouldn't apply. I made a couple of changes to the part that was rejected and it applied, tried "ant test" in solr/, and it failed. It looks like the 3.2 code is missing some of the bits that say "<Object>" ... not knowing Java, I have no idea what this might mean.

          Show
          Shawn Heisey added a comment - Thank you, Adrien! I tried to apply this to Solr 3.2, which I am in the process of rolling into production, but it wouldn't apply. I made a couple of changes to the part that was rejected and it applied, tried "ant test" in solr/, and it failed. It looks like the 3.2 code is missing some of the bits that say "<Object>" ... not knowing Java, I have no idea what this might mean.
          Hide
          Adrien Grand added a comment -

          Shawn,

          This patch was built on a recent trunk checkout, which explains why it didn't apply on a 3.x version: the getStatistics of 3.x works with NamedList while the same method in 4.x works with NamedList<Object>.

          I updated the patch to fix a little bug when the statistics were requested on the same millisecond as the request handler started.

          I also removed a few modifications I had made to remove compiler warnings. Although desirable, it should probably go to a separate commit. This makes the patch a little bit clearer too.

          I've just run Solr tests again with the new version of the patch and I don't get any failure/error.

          Show
          Adrien Grand added a comment - Shawn, This patch was built on a recent trunk checkout, which explains why it didn't apply on a 3.x version: the getStatistics of 3.x works with NamedList while the same method in 4.x works with NamedList<Object>. I updated the patch to fix a little bug when the statistics were requested on the same millisecond as the request handler started. I also removed a few modifications I had made to remove compiler warnings. Although desirable, it should probably go to a separate commit. This makes the patch a little bit clearer too. I've just run Solr tests again with the new version of the patch and I don't get any failure/error.
          Hide
          Shawn Heisey added a comment -

          I took your new patch, removed all instances of "<Object>" from it, and applied it to 3.2. It went right in, and all tests passed. I also added a 95th percentile, and I'm recompiling. After that I'm going to put it in place and give it a try.

          Show
          Shawn Heisey added a comment - I took your new patch, removed all instances of "<Object>" from it, and applied it to 3.2. It went right in, and all tests passed. I also added a 95th percentile, and I'm recompiling. After that I'm going to put it in place and give it a try.
          Hide
          Adrien Grand added a comment -

          Improved version of the patch :

          • warning + default values instead of exceptions in initArgs,
          • configurable list of percentiles to compute (defaulting to the 90th and 99th percentiles).
          Show
          Adrien Grand added a comment - Improved version of the patch : warning + default values instead of exceptions in initArgs, configurable list of percentiles to compute (defaulting to the 90th and 99th percentiles).
          Hide
          Shawn Heisey added a comment -

          The same removal of "<Object>" from the newest patch before applying it makes it work on 3.2. This is really awesome. Thank you! I hope it will be committed to 3.2 and branch_3x as well as trunk.

          Show
          Shawn Heisey added a comment - The same removal of "<Object>" from the newest patch before applying it makes it work on 3.2. This is really awesome. Thank you! I hope it will be committed to 3.2 and branch_3x as well as trunk.
          Hide
          Shawn Heisey added a comment -

          I noticed what might be a problem. Config:

          <lst name="rollingStatistics">
          <int name="history">604800</int>
          <int name="samples">16384</int>
          <arr name="percentiles">
          <int>90</int>
          <int>95</int>
          <int>99</int>
          </arr>
          </lst>

          Current statistics generated by Solr 3.2 with this patch:

          requests : 31861
          errors : 0
          timeouts : 0
          totalTime : 454186
          avgTimePerRequest : 14.255
          rollingRequests : 16384
          rollingTotalTime : 106955
          rollingAvgTimePerRequest : 6.528
          rollingAvgRequestsPerSecond : -0.024
          rollingMedian : 1
          rolling90thPercentile : 9
          rolling95thPercentile : 20
          rolling99thPercentile : 96

          The requests per second is negative, which I think is a clear problem.

          I am slightly concerned by seeing the median as 1 millisecond. The shard has 9 million documents in it and the index is 17GB, with only 9GB of RAM in the VM. It might not actually be a problem – I was hitting it several times with a homegrown benchmarking app that has a relatively limited (only a couple thousand) supply of search strings. It's entirely possible that after the first few thousand queries, everything was being served from Solr caches, and that what's still in the rolling data store really did happen that quickly.

          Show
          Shawn Heisey added a comment - I noticed what might be a problem. Config: <lst name="rollingStatistics"> <int name="history">604800</int> <int name="samples">16384</int> <arr name="percentiles"> <int>90</int> <int>95</int> <int>99</int> </arr> </lst> Current statistics generated by Solr 3.2 with this patch: requests : 31861 errors : 0 timeouts : 0 totalTime : 454186 avgTimePerRequest : 14.255 rollingRequests : 16384 rollingTotalTime : 106955 rollingAvgTimePerRequest : 6.528 rollingAvgRequestsPerSecond : -0.024 rollingMedian : 1 rolling90thPercentile : 9 rolling95thPercentile : 20 rolling99thPercentile : 96 The requests per second is negative, which I think is a clear problem. I am slightly concerned by seeing the median as 1 millisecond. The shard has 9 million documents in it and the index is 17GB, with only 9GB of RAM in the VM. It might not actually be a problem – I was hitting it several times with a homegrown benchmarking app that has a relatively limited (only a couple thousand) supply of search strings. It's entirely possible that after the first few thousand queries, everything was being served from Solr caches, and that what's still in the rolling data store really did happen that quickly.
          Hide
          Adrien Grand added a comment -

          The median response time looks good compared to the 90th, 95th and 99th percentiles. Your explanation with Solr caches sounds reasonable to me.

          Regarding the value of rollingAvgRequestsPerSecond, I suspect it is negative because of the time span between the computation of 'now' and the time when rolling data is used. I will try to provide a patch for this issue.

          Show
          Adrien Grand added a comment - The median response time looks good compared to the 90th, 95th and 99th percentiles. Your explanation with Solr caches sounds reasonable to me. Regarding the value of rollingAvgRequestsPerSecond, I suspect it is negative because of the time span between the computation of 'now' and the time when rolling data is used. I will try to provide a patch for this issue.
          Hide
          Shawn Heisey added a comment -

          A little further info on rollingAvgRequestsPerSecond ... I have noticed that it is always different from AvgRequestsPerSecond, even when requests and rollingRequests are the same. I would expect different numbers when requests and rollingRequests diverge, but not when they are the same.

          I did take a look at the code, but have to admit that I haven't wrapped my brain around it enough to figure out what the problem might be.

          Show
          Shawn Heisey added a comment - A little further info on rollingAvgRequestsPerSecond ... I have noticed that it is always different from AvgRequestsPerSecond, even when requests and rollingRequests are the same. I would expect different numbers when requests and rollingRequests diverge, but not when they are the same. I did take a look at the code, but have to admit that I haven't wrapped my brain around it enough to figure out what the problem might be.
          Hide
          Adrien Grand added a comment -

          Hi Shawn,

          I fixed the patch so that rolling statistics are now consistent with non-rolling statistics for the first requests. Average requests by second may sometimes be a little different, but ensuring rolling and non-rolling statistics have exactly the same value would require more synchronization, which is not an option in my opinion.

          Please let me know if you still get negative values for rollingAvgRequestsPerSecond with this patch.

          I hope this patch is the good one!

          Show
          Adrien Grand added a comment - Hi Shawn, I fixed the patch so that rolling statistics are now consistent with non-rolling statistics for the first requests. Average requests by second may sometimes be a little different, but ensuring rolling and non-rolling statistics have exactly the same value would require more synchronization, which is not an option in my opinion. Please let me know if you still get negative values for rollingAvgRequestsPerSecond with this patch. I hope this patch is the good one!
          Hide
          Shawn Heisey added a comment -

          I had a stray thought, wondering whether you could eliminate ":" queries from the rollingStatistics. I then realized that if you set low and high responseTime thresholds, you could effectively do the same thing.

          I've cooked up a patch to do exactly this. Although I won't be using highThreshold myself, I figured someone might want it, so I put it in there.

          I created this against 3.2. It should apply to 3.3 as well. I manually modified the NamedList entries back to NamedList<?> and created a second patch, with the theory that it would then apply against trunk correctly. I'll attach both patch versions.

          Show
          Shawn Heisey added a comment - I had a stray thought, wondering whether you could eliminate " : " queries from the rollingStatistics. I then realized that if you set low and high responseTime thresholds, you could effectively do the same thing. I've cooked up a patch to do exactly this. Although I won't be using highThreshold myself, I figured someone might want it, so I put it in there. I created this against 3.2. It should apply to 3.3 as well. I manually modified the NamedList entries back to NamedList<?> and created a second patch, with the theory that it would then apply against trunk correctly. I'll attach both patch versions.
          Hide
          Shawn Heisey added a comment -

          New patches for 3.2/3.3, and a patch for trunk. It adds lowThreshold and highThreshold config options. If I broke some cardinal Java rule in my changes, please feel free to fix!

          Show
          Shawn Heisey added a comment - New patches for 3.2/3.3, and a patch for trunk. It adds lowThreshold and highThreshold config options. If I broke some cardinal Java rule in my changes, please feel free to fix!
          Hide
          Shawn Heisey added a comment -

          I tried to add a percentile of 100, so I could see the slowest query, and it didn't seem to do anything. I'll be changing the following line so it works:

          if (percentile >= 0 && percentile < 100) {

          Show
          Shawn Heisey added a comment - I tried to add a percentile of 100, so I could see the slowest query, and it didn't seem to do anything. I'll be changing the following line so it works: if (percentile >= 0 && percentile < 100) {
          Hide
          Shawn Heisey added a comment -

          Of course, adding support for a 100th percentile was NOT as easy as simply changing < to <=. New patches.

          Show
          Shawn Heisey added a comment - Of course, adding support for a 100th percentile was NOT as easy as simply changing < to <=. New patches.
          Hide
          Hoss Man added a comment -

          Shawn: i was a little concerned based on the issue summary that this would slow down the performance of the handlers, but skimming the patch i see that unless it's explicitly configured in the init params, only a single "responseTimes != null" check is added to each request, and even when it is enabled, all the hard work is defered until the stats are actually requested – so as long as we document that recording (and retrieving) these stats may slow down the performance of hte handler, i don't see a big problem there. (my personal prefrence is to generate these stats from log parsing because then it can be done completely out of band – but i certainly won't object to making it easier for people to get directly from solr)

          my only other suggestion is that you refactor the stats code into some static methods (or a helper class) so we can have some test cases that verify the accuracy against fixed data ... i'm not a match guy, so it's not entirely obvious to me just looking at it that it will always produce the correct results (particularly with the max age and array re-ordering and what not) so unit tests would help demonstrate tht it does what it's suppose to in various edge cases.

          Show
          Hoss Man added a comment - Shawn: i was a little concerned based on the issue summary that this would slow down the performance of the handlers, but skimming the patch i see that unless it's explicitly configured in the init params, only a single "responseTimes != null" check is added to each request, and even when it is enabled, all the hard work is defered until the stats are actually requested – so as long as we document that recording (and retrieving) these stats may slow down the performance of hte handler, i don't see a big problem there. (my personal prefrence is to generate these stats from log parsing because then it can be done completely out of band – but i certainly won't object to making it easier for people to get directly from solr) my only other suggestion is that you refactor the stats code into some static methods (or a helper class) so we can have some test cases that verify the accuracy against fixed data ... i'm not a match guy, so it's not entirely obvious to me just looking at it that it will always produce the correct results (particularly with the max age and array re-ordering and what not) so unit tests would help demonstrate tht it does what it's suppose to in various edge cases.
          Hide
          Shawn Heisey added a comment -

          Hoss, the patch isn't my work, I just modified it to support a 100th percentile and reattached it. I am only just now beginning to learn Java. Although I have some clue what you're saying with static methods, actually doing it properly within a larger work like Solr is something I won't be able to do yet.

          Show
          Shawn Heisey added a comment - Hoss, the patch isn't my work, I just modified it to support a 100th percentile and reattached it. I am only just now beginning to learn Java. Although I have some clue what you're saying with static methods, actually doing it properly within a larger work like Solr is something I won't be able to do yet.
          Hide
          Shawn Heisey added a comment -

          Based on filenames, I couldn't find an existing unit test that checks handler statistics, so I couldn't figure out how to make a test for this patch.

          I am very interested in getting this included in branch_3x. If you have some example code I can look at to create unit tests, I can look into making one.

          Show
          Shawn Heisey added a comment - Based on filenames, I couldn't find an existing unit test that checks handler statistics, so I couldn't figure out how to make a test for this patch. I am very interested in getting this included in branch_3x. If you have some example code I can look at to create unit tests, I can look into making one.
          Hide
          Eric Pugh added a comment -

          Has anyone had thoughts on how to do this via a component that is less intrusive then modifying RequestHandlerBase? I'd love to do this via a component that I could compile as a standalone project and then drop in my existing Solr.

          Also, I am only interested in certain subset of queries, so I added a collection of regex patterns as that are used to test against the query string to see if it should be included in the rolling statistics. I will upload the patch. Also fixed the patch to work against the latest trunk.

          Show
          Eric Pugh added a comment - Has anyone had thoughts on how to do this via a component that is less intrusive then modifying RequestHandlerBase? I'd love to do this via a component that I could compile as a standalone project and then drop in my existing Solr. Also, I am only interested in certain subset of queries, so I added a collection of regex patterns as that are used to test against the query string to see if it should be included in the rolling statistics. I will upload the patch. Also fixed the patch to work against the latest trunk.
          Hide
          Eric Pugh added a comment -

          Updated to latest trunk, added regex patterns.

          Show
          Eric Pugh added a comment - Updated to latest trunk, added regex patterns.
          Hide
          Shawn Heisey added a comment -

          Attaching a branch_3x version of Eric's latest patch. If applied to 3.5, all tests pass. A current checkout of branch_3x without this patch applied fails dataimport handler tests, adding the patch does not introduce any additional failures.

          Show
          Shawn Heisey added a comment - Attaching a branch_3x version of Eric's latest patch. If applied to 3.5, all tests pass. A current checkout of branch_3x without this patch applied fails dataimport handler tests, adding the patch does not introduce any additional failures.
          Hide
          Shawn Heisey added a comment -

          Let's have everyone pretend for a minute that I have a slightly better grasp of Solr/Lucene internals than I actually do. Perhaps I will one day be able to take what you say and figure out what it means. I am very interested in having these stats available without patching Solr on my own.

          What would be the right way to go about re-implementing this as a module (along with some unit tests) so the code could be committed?

          Show
          Shawn Heisey added a comment - Let's have everyone pretend for a minute that I have a slightly better grasp of Solr/Lucene internals than I actually do. Perhaps I will one day be able to take what you say and figure out what it means. I am very interested in having these stats available without patching Solr on my own. What would be the right way to go about re-implementing this as a module (along with some unit tests) so the code could be committed?
          Hide
          Shawn Heisey added a comment -

          Purely hypothetical stuff, probably way beyond my skills: Would it be possible (and useful) to use a Lucene index (RAMDirectory maybe) to store the query time data for performance reasons, or is the current array implementation good enough?

          Show
          Shawn Heisey added a comment - Purely hypothetical stuff, probably way beyond my skills: Would it be possible (and useful) to use a Lucene index (RAMDirectory maybe) to store the query time data for performance reasons, or is the current array implementation good enough?
          Hide
          Alan Woodward added a comment -

          I wonder if Coda Hale's Metrics library might be worth using here?

          http://metrics.codahale.com/

          It already deals with rolling updates, and can expose measurements through JMX beans.

          Show
          Alan Woodward added a comment - I wonder if Coda Hale's Metrics library might be worth using here? http://metrics.codahale.com/ It already deals with rolling updates, and can expose measurements through JMX beans.
          Hide
          Shawn Heisey added a comment -

          I wonder if Coda Hale's Metrics library might be worth using here?

          That does look really interesting. Perhaps a bit above my skill set, and it would bloat the .war somewhat. It would provide the opportunity to record metrics all over the place, not just the things we originally cooked up here.

          In the meantime, I have updated the current patch to branch_4x.

          Show
          Shawn Heisey added a comment - I wonder if Coda Hale's Metrics library might be worth using here? That does look really interesting. Perhaps a bit above my skill set, and it would bloat the .war somewhat. It would provide the opportunity to record metrics all over the place, not just the things we originally cooked up here. In the meantime, I have updated the current patch to branch_4x.
          Hide
          Shawn Heisey added a comment -

          An updated patch against branch_4x. I massaged the solrconfig.xml options a bit trying to make them a bit easier to understand. I also fixed all the statistics reporting glitches I could find.

          Show
          Shawn Heisey added a comment - An updated patch against branch_4x. I massaged the solrconfig.xml options a bit trying to make them a bit easier to understand. I also fixed all the statistics reporting glitches I could find.
          Hide
          Alan Woodward added a comment -

          > and it would bloat the .war somewhat

          metrics-core is only 81kb And its only dependency is slf4j-api, which is already included.

          I think this might be an easy win, actually. Replacing the requests, errors and timeouts counts with Counters, and adding a TimerContext reproduces all the existing stats and gives you the histogram of request times. Plus it's extensible, as you say.

          Show
          Alan Woodward added a comment - > and it would bloat the .war somewhat metrics-core is only 81kb And its only dependency is slf4j-api, which is already included. I think this might be an easy win, actually. Replacing the requests, errors and timeouts counts with Counters, and adding a TimerContext reproduces all the existing stats and gives you the histogram of request times. Plus it's extensible, as you say.
          Hide
          Shawn Heisey added a comment -

          New patch against branch_4x. The code hasn't changed, just the example solrconfig.xml. The config is now commented out and says EXPERIMENTAL. No performance impact should occur when the config is not used.

          The metrics package Alan mentioned looks like a better long-term solution, but in the meantime, we have this.

          It hasn't received widespread testing, but I've been using a previous version of the patch in 3.x for a very long time (16384 samples on two handlers) with no trouble. Our query volume is low, so we have never noticed any performance impact. It's provided a lot of valuable insight into Solr performance. I think a lot of other people would get value out of this.

          Show
          Shawn Heisey added a comment - New patch against branch_4x. The code hasn't changed, just the example solrconfig.xml. The config is now commented out and says EXPERIMENTAL. No performance impact should occur when the config is not used. The metrics package Alan mentioned looks like a better long-term solution, but in the meantime, we have this. It hasn't received widespread testing, but I've been using a previous version of the patch in 3.x for a very long time (16384 samples on two handlers) with no trouble. Our query volume is low, so we have never noticed any performance impact. It's provided a lot of valuable insight into Solr performance. I think a lot of other people would get value out of this.
          Hide
          Alan Woodward added a comment -

          Here's a patch that uses the metrics library. It doesn't include Eric's regex matching or anything at the moment - it basically just takes what's currently in trunk, refactors it to use metrics' Counter and Timer objects, and adds the rolling average data.

          Cons:

          • it adds another dependency to solr-core. It's a useful dependency, IMO, but still.
          • tests don't pass at the moment, as metrics spawns extra threads which the test runner doesn't know how to deal with

          Pros:

          • it's a purpose-designed stats and metrics library, so we don't need to worry about the maths or sampling algorithms
          • it adds the functionality of the original ticket/patch in a much simpler way.

          The ideal solution would be a component of some kind, I think, but this at least improves on what's in trunk at the moment.

          Show
          Alan Woodward added a comment - Here's a patch that uses the metrics library. It doesn't include Eric's regex matching or anything at the moment - it basically just takes what's currently in trunk, refactors it to use metrics' Counter and Timer objects, and adds the rolling average data. Cons: it adds another dependency to solr-core. It's a useful dependency, IMO, but still. tests don't pass at the moment, as metrics spawns extra threads which the test runner doesn't know how to deal with Pros: it's a purpose-designed stats and metrics library, so we don't need to worry about the maths or sampling algorithms it adds the functionality of the original ticket/patch in a much simpler way. The ideal solution would be a component of some kind, I think, but this at least improves on what's in trunk at the moment.
          Hide
          Shawn Heisey added a comment -

          Awesome, Alan! What options might we have to prevent long-running handlers from accumulating huge metrics histories and chewing up tons of RAM? Is there a get75thpercentile method? With the old patch, I do 75, 95, and 99. I would also like to add 99.9, but the old patch uses ints so that wasn't possible.

          When I have a moment, I will attempt to look at the javadocs for the package and answer my own questions. Unless you get to it first, I will also attempt to mod the patch to expose any memory-limiting options.

          Show
          Shawn Heisey added a comment - Awesome, Alan! What options might we have to prevent long-running handlers from accumulating huge metrics histories and chewing up tons of RAM? Is there a get75thpercentile method? With the old patch, I do 75, 95, and 99. I would also like to add 99.9, but the old patch uses ints so that wasn't possible. When I have a moment, I will attempt to look at the javadocs for the package and answer my own questions. Unless you get to it first, I will also attempt to mod the patch to expose any memory-limiting options.
          Hide
          Alan Woodward added a comment -

          Hi Shawn,

          Metrics uses reservoir sampling to maintain its measurements, so the history is actually always a fixed size. This is configurable, but defaults to 1024 entries. There's more information at http://metrics.codahale.com/manual/core/#histograms and http://www.johndcook.com/standard_deviation.html.

          There are get75thpercentile and get999thpercentile methods out of the box, and you can also ask for values at arbitrary percentiles using getValue().

          Show
          Alan Woodward added a comment - Hi Shawn, Metrics uses reservoir sampling to maintain its measurements, so the history is actually always a fixed size. This is configurable, but defaults to 1024 entries. There's more information at http://metrics.codahale.com/manual/core/#histograms and http://www.johndcook.com/standard_deviation.html . There are get75thpercentile and get999thpercentile methods out of the box, and you can also ask for values at arbitrary percentiles using getValue().
          Hide
          Shawn Heisey added a comment -

          I have answers to some of my questions. There is a 75th percentile. I added the 75th and 999th to what you had, and it seems to display the stats page a lot faster than my patch did. We'll see what happens when it gets a few thousand queries under its belt, though. I was running the old patch with 16384 samples, and I put the stats on three handlers, so it was having to copy arrays of 16384 longs a total of six times every time I refreshed the stats page. I may also add the 98th percentile. It may be a good idea to make each percentile point configurable in solrconfig.xml. So far I have not yet figured out whether it is possible to limit the number of samples stored, or anything else which can limit the amount of memory required.

          The names for the average req/s over the last 5 and 15 minutes are REALLY long. Unless you have a high res display (1920 pixels wide) and maximize the window, the names overlap the values. If I think of a reasonable way to shorten those, I will. I ran into it myself when making my branch_4x patch.

          Show
          Shawn Heisey added a comment - I have answers to some of my questions. There is a 75th percentile. I added the 75th and 999th to what you had, and it seems to display the stats page a lot faster than my patch did. We'll see what happens when it gets a few thousand queries under its belt, though. I was running the old patch with 16384 samples, and I put the stats on three handlers, so it was having to copy arrays of 16384 longs a total of six times every time I refreshed the stats page. I may also add the 98th percentile. It may be a good idea to make each percentile point configurable in solrconfig.xml. So far I have not yet figured out whether it is possible to limit the number of samples stored, or anything else which can limit the amount of memory required. The names for the average req/s over the last 5 and 15 minutes are REALLY long. Unless you have a high res display (1920 pixels wide) and maximize the window, the names overlap the values. If I think of a reasonable way to shorten those, I will. I ran into it myself when making my branch_4x patch.
          Hide
          Shawn Heisey added a comment -

          I didn't see your reply about the reservoir size until after I'd already submitted mine. If I want to increase/decrease that size, how do I do that? So far poking around the javadocs and using google hasn't turned anything up.

          Show
          Shawn Heisey added a comment - I didn't see your reply about the reservoir size until after I'd already submitted mine. If I want to increase/decrease that size, how do I do that? So far poking around the javadocs and using google hasn't turned anything up.
          Hide
          Alan Woodward added a comment -

          It looks like the sample size for Timer objects is hard-wired.

          It's only 8kB per Timer, though, which is pretty much a rounding error when compared with Solr's memory usage.

          Show
          Alan Woodward added a comment - It looks like the sample size for Timer objects is hard-wired. It's only 8kB per Timer, though, which is pretty much a rounding error when compared with Solr's memory usage.
          Hide
          Shawn Heisey added a comment -

          Originally I was worried about it being too large, because I didn't see anything defining the size. Now, knowing that it's only 1024, I consider it to be way too small. I was using 16384 for the old patch, I was actually hoping to make it a bit larger than that.

          Show
          Shawn Heisey added a comment - Originally I was worried about it being too large, because I didn't see anything defining the size. Now, knowing that it's only 1024, I consider it to be way too small. I was using 16384 for the old patch, I was actually hoping to make it a bit larger than that.
          Hide
          Shawn Heisey added a comment -

          After poking around a lot looking for a way to bump the reservoir size, I finally came across the paper on reservoir sampling by Vitter. After even more poking around, I think I get it now. Their small reservoir apparently really does give statistically relevant results over millions or billions of total samples. If it didn't give them numbers they could use, they would have already made it larger.

          Do you think it's worthwhile to give people the ability to customize the percentile list – turn some of the standard percentiles off, and/or add custom ones? As soon as we conclude that including the full predefined set won't present a performance problem because it only gets calculated when the admin GUI is accessed, there'll be someone who has created hundreds of request handlers and polls the statistics for all of them once a minute. I can also see someone wanting to see the 12th and 87th percentiles for some reason neither of us can fathom, but makes perfect sense to them.

          Show
          Shawn Heisey added a comment - After poking around a lot looking for a way to bump the reservoir size, I finally came across the paper on reservoir sampling by Vitter. After even more poking around, I think I get it now. Their small reservoir apparently really does give statistically relevant results over millions or billions of total samples. If it didn't give them numbers they could use, they would have already made it larger. Do you think it's worthwhile to give people the ability to customize the percentile list – turn some of the standard percentiles off, and/or add custom ones? As soon as we conclude that including the full predefined set won't present a performance problem because it only gets calculated when the admin GUI is accessed, there'll be someone who has created hundreds of request handlers and polls the statistics for all of them once a minute. I can also see someone wanting to see the 12th and 87th percentiles for some reason neither of us can fathom, but makes perfect sense to them.
          Hide
          Alan Woodward added a comment -

          New patch, adding 75th and 999th percentile, making the stats names less insanely long, and adding the metrics- threads to the test excluder thingy. All solr-core tests pass.

          Show
          Alan Woodward added a comment - New patch, adding 75th and 999th percentile, making the stats names less insanely long, and adding the metrics- threads to the test excluder thingy. All solr-core tests pass.
          Hide
          Alan Woodward added a comment -

          I think we'd probably want to keep the percentile list limited to start with. People can always ask for improvements later if they need them.

          I think this is ready to go in?

          Show
          Alan Woodward added a comment - I think we'd probably want to keep the percentile list limited to start with. People can always ask for improvements later if they need them. I think this is ready to go in?
          Hide
          Stefan Matheis (steffkes) added a comment -

          Hey, i applied to Patch to see where the Output goes and how it looks in the admin GUI .. right now, it's listed in the stats-section, as a table with all the other given attributes. how helpful would it be to have some kind of graph here? perhaps one like we have already on the dashboard to see the memory-usage? Let me know what you think about it

          Show
          Stefan Matheis (steffkes) added a comment - Hey, i applied to Patch to see where the Output goes and how it looks in the admin GUI .. right now, it's listed in the stats-section, as a table with all the other given attributes. how helpful would it be to have some kind of graph here? perhaps one like we have already on the dashboard to see the memory-usage? Let me know what you think about it
          Hide
          Alan Woodward added a comment -

          Hey Stefan, yeah I've been thinking about graphs, pictures are always good. For this sort of information, though, I think you generally want time-series representations, and for that you want proper monitoring software (something like graphite). So I think the best thing we can do here is just expose the data, and let people plug in their own monitors. Unless you have a better idea for how to represent it?

          Show
          Alan Woodward added a comment - Hey Stefan, yeah I've been thinking about graphs, pictures are always good. For this sort of information, though, I think you generally want time-series representations, and for that you want proper monitoring software (something like graphite). So I think the best thing we can do here is just expose the data, and let people plug in their own monitors. Unless you have a better idea for how to represent it?
          Hide
          Shawn Heisey added a comment - - edited

          This is lightyears beyond what I could have hoped for when I first opened this issue. I do have one more picky note: the extreme floating point precision of the output. Here's what I am getting:

          handlerStart: 1351011529717
          requests: 14
          errors: 0
          timeouts: 0
          totalTime: 53483.059463
          avgTimePerRequest: 3820.2185330714287
          avgRequestsPerSecond: 0.1592740558481214
          5minRateReqsPerSecond: 0.6393213424767414
          15minRateReqsPerSecond: 0.7422605686168207
          medianRequestTime: 2537.401157
          75thPcRequestTime: 7728.151086
          95thPcRequestTime: 8963.867643
          99thPcRequestTime: 8963.867643
          999thPcRequestTime: 8963.867643

          Here's what I would like to see instead ... lower precision, and rounded up when the first eliminated digit is 5 or higher.

          handlerStart: 1351011529717
          requests: 14
          errors: 0
          timeouts: 0
          totalTime: 53483
          avgTimePerRequest: 3820.22
          avgRequestsPerSecond: 0.16
          5minRateReqsPerSecond: 0.64
          15minRateReqsPerSecond: 0.74
          medianRequestTime: 2537.40
          75thPcRequestTime: 7728.15
          95thPcRequestTime: 8963.87
          99thPcRequestTime: 8963.87
          999thPcRequestTime: 8963.87

          Show
          Shawn Heisey added a comment - - edited This is lightyears beyond what I could have hoped for when I first opened this issue. I do have one more picky note: the extreme floating point precision of the output. Here's what I am getting: handlerStart: 1351011529717 requests: 14 errors: 0 timeouts: 0 totalTime: 53483.059463 avgTimePerRequest: 3820.2185330714287 avgRequestsPerSecond: 0.1592740558481214 5minRateReqsPerSecond: 0.6393213424767414 15minRateReqsPerSecond: 0.7422605686168207 medianRequestTime: 2537.401157 75thPcRequestTime: 7728.151086 95thPcRequestTime: 8963.867643 99thPcRequestTime: 8963.867643 999thPcRequestTime: 8963.867643 Here's what I would like to see instead ... lower precision, and rounded up when the first eliminated digit is 5 or higher. handlerStart: 1351011529717 requests: 14 errors: 0 timeouts: 0 totalTime: 53483 avgTimePerRequest: 3820.22 avgRequestsPerSecond: 0.16 5minRateReqsPerSecond: 0.64 15minRateReqsPerSecond: 0.74 medianRequestTime: 2537.40 75thPcRequestTime: 7728.15 95thPcRequestTime: 8963.87 99thPcRequestTime: 8963.87 999thPcRequestTime: 8963.87
          Hide
          Shawn Heisey added a comment -

          I would also move avgTimePerRequest to right before medianRequestTime so similar numbers are all together. I was going to suggest giving it a new name to jive with the additions, but there are likely a lot of existing customer scripts that rely on that name – including some of mine.

          Show
          Shawn Heisey added a comment - I would also move avgTimePerRequest to right before medianRequestTime so similar numbers are all together. I was going to suggest giving it a new name to jive with the additions, but there are likely a lot of existing customer scripts that rely on that name – including some of mine.
          Hide
          Shawn Heisey added a comment - - edited

          I have now discovered a real problem. All of my search handlers have the exact same statistics. I have defined /lbcheck, /ncdismax, /search, and /select.

          That brings to mind a potential test that could be added – make sure that when you issue queries against one handler, stats on other handlers do not see their numbers go up. No idea how to write it, though.

          Show
          Shawn Heisey added a comment - - edited I have now discovered a real problem. All of my search handlers have the exact same statistics. I have defined /lbcheck, /ncdismax, /search, and /select. That brings to mind a potential test that could be added – make sure that when you issue queries against one handler, stats on other handlers do not see their numbers go up. No idea how to write it, though.
          Hide
          Alan Woodward added a comment -

          Hm, OK. I'm creating the various Metrics objects in the base class constructor, and registering them by class using this.getClass(). Only problem here is that in a super constructor, getClass() returns the superclass. Oops.

          If I move the object creation to init() I get other errors, because RequestHandlers are registered with JMX before init() is called, and JMX calls getStatistics() to get all the various measurement names and register them.

          Maybe put a guard in getStatistics to check if the counters are null, and if they are, instantiate them? Seems a bit hacky though. Let me have a think about this.

          In re the precision of the measurements, the jscript in the front end could presumably round them to 2 sig figs - that way they look prettier in the UI, but are still precise for any client that wants to use it.

          Show
          Alan Woodward added a comment - Hm, OK. I'm creating the various Metrics objects in the base class constructor, and registering them by class using this.getClass(). Only problem here is that in a super constructor, getClass() returns the superclass. Oops. If I move the object creation to init() I get other errors, because RequestHandlers are registered with JMX before init() is called, and JMX calls getStatistics() to get all the various measurement names and register them. Maybe put a guard in getStatistics to check if the counters are null, and if they are, instantiate them? Seems a bit hacky though. Let me have a think about this. In re the precision of the measurements, the jscript in the front end could presumably round them to 2 sig figs - that way they look prettier in the UI, but are still precise for any client that wants to use it.
          Hide
          Shawn Heisey added a comment -

          I think I fixed it.

          It looks like if you pass the same combination of arguments to the newCounter/newTimer methods, you actually get back the same object as the last time it was called with those parameters, not a new one. There is an alternate form of the constructor that takes a "scope" argument. I could have appended the new value to the name argument, but since they were kind enough to provide something separate...

          Show
          Shawn Heisey added a comment - I think I fixed it. It looks like if you pass the same combination of arguments to the newCounter/newTimer methods, you actually get back the same object as the last time it was called with those parameters, not a new one. There is an alternate form of the constructor that takes a "scope" argument. I could have appended the new value to the name argument, but since they were kind enough to provide something separate...
          Hide
          Shawn Heisey added a comment -

          I never know when things are threadsafe, although it does seem to work. Does a static class member variable 'automatically' become threadsafe, or would volatile be required? Was protected the right way to do that?

          Show
          Shawn Heisey added a comment - I never know when things are threadsafe, although it does seem to work. Does a static class member variable 'automatically' become threadsafe, or would volatile be required? Was protected the right way to do that?
          Hide
          Otis Gospodnetic added a comment -

          Alan Woodward are these metrics being published in JMX? They should, and Code's Metrics should make that easy. I had 2.5 second look at the patch and didn't find any mentions of "jmx".

          Show
          Otis Gospodnetic added a comment - Alan Woodward are these metrics being published in JMX? They should, and Code's Metrics should make that easy. I had 2.5 second look at the patch and didn't find any mentions of "jmx".
          Hide
          Shawn Heisey added a comment -

          Solr handler statistics were already published in JMX, this just added some entries. I pulled up jconsole and connected to my patched solr server and the new stats were there.

          Show
          Shawn Heisey added a comment - Solr handler statistics were already published in JMX, this just added some entries. I pulled up jconsole and connected to my patched solr server and the new stats were there.
          Hide
          Alan Woodward added a comment -

          Updated patch, using this.toString() as the scope identifier. Your handlerCount solution wouldn't have been thread-safe, Shawn, but thanks for finding the right method to use!

          Also adds a test to check that different handlers have different statistics.

          Otis Gospodnetic as Shawn says, this is just extending the existing mbeans, so it's already available through JMX. Metrics also exposes everything through JMX by default anyway, so you can get the stats either way.

          Show
          Alan Woodward added a comment - Updated patch, using this.toString() as the scope identifier. Your handlerCount solution wouldn't have been thread-safe, Shawn, but thanks for finding the right method to use! Also adds a test to check that different handlers have different statistics. Otis Gospodnetic as Shawn says, this is just extending the existing mbeans, so it's already available through JMX. Metrics also exposes everything through JMX by default anyway, so you can get the stats either way.
          Hide
          Shawn Heisey added a comment -

          I wanted to double-check that the new test would fail with the old constructor, so I installed your new patch and removed this.toString() from the parameter list on those statements.

          The updated RequestHandlerTests didn't fail like I'd hoped. I suspect that's because the terms handler and the update handler are different classes, so this.getClass() was apparently different. Before the fix, I did have different statistics in the update handler versus the search handlers. It was only handlers of the same type that were the same – my four search handlers.

          I did however get a rather spectacular (and repeatable) failure in MBeansHandlerTest. That failure went away when I restored the constructor to the current patch state. When I make it into the office, I will do some additional testing to make sure I did everything right.

          Show
          Shawn Heisey added a comment - I wanted to double-check that the new test would fail with the old constructor, so I installed your new patch and removed this.toString() from the parameter list on those statements. The updated RequestHandlerTests didn't fail like I'd hoped. I suspect that's because the terms handler and the update handler are different classes, so this.getClass() was apparently different. Before the fix, I did have different statistics in the update handler versus the search handlers. It was only handlers of the same type that were the same – my four search handlers. I did however get a rather spectacular (and repeatable) failure in MBeansHandlerTest. That failure went away when I restored the constructor to the current patch state. When I make it into the office, I will do some additional testing to make sure I did everything right.
          Hide
          Shawn Heisey added a comment - - edited

          With this as the contructor:

            public RequestHandlerBase() {
          //    numRequests = Metrics.newCounter(RequestHandlerBase.class, "numRequests", this.toString());
          //    numErrors = Metrics.newCounter(RequestHandlerBase.class, "numErrors", this.toString());
          //    numTimeouts = Metrics.newCounter(RequestHandlerBase.class, "numTimeouts", this.toString());
          //    requestTimes = Metrics.newTimer(RequestHandlerBase.class, "requestTimes", this.toString());
              numRequests = Metrics.newCounter(RequestHandlerBase.class, "numRequests");
              numErrors = Metrics.newCounter(RequestHandlerBase.class, "numErrors");
              numTimeouts = Metrics.newCounter(RequestHandlerBase.class, "numTimeouts");
              requestTimes = Metrics.newTimer(RequestHandlerBase.class, "requestTimes");
            }
          

          I get the following as the failure. I suppose I should be glad that it fails, but RequestHandlersTests, which is the test that was modified, continues to pass.

          [junit4:junit4] Suite: org.apache.solr.handler.admin.MBeansHandlerTest
          [junit4:junit4]   2> 0 T297 oas.SolrTestCaseJ4.initCore ####initCore
          [junit4:junit4]   2> Creating dataDir: /index/src/branch_4x/solr/build/solr-core/test/J0/./solrtest-MBeansHandlerTest-1351102494087
          [junit4:junit4]   2> 26 T297 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/index/src/branch_4x/solr/build/solr-core/test-files/solr/collection1/'
          [junit4:junit4]   2> 27 T297 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/index/src/branch_4x/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
          [junit4:junit4]   2> 27 T297 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/index/src/branch_4x/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
          [junit4:junit4]   2> 59 T297 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
          [junit4:junit4]   2> 98 T297 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
          [junit4:junit4]   2> 98 T297 oass.IndexSchema.readSchema Reading Solr Schema
          [junit4:junit4]   2> 105 T297 oass.IndexSchema.readSchema Schema name=test
          [junit4:junit4]   2> 414 T297 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
          [junit4:junit4]   2> 420 T297 oass.IndexSchema.readSchema default search field in schema is text
          [junit4:junit4]   2> 422 T297 oass.IndexSchema.readSchema unique key field: id
          [junit4:junit4]   2> 430 T297 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
          [junit4:junit4]   2> 432 T297 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
          [junit4:junit4]   2> 434 T297 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
          [junit4:junit4]   2> 435 T297 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
          [junit4:junit4]   2> 436 T297 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
          [junit4:junit4]   2> 436 T297 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /index/src/branch_4x/solr/build/solr-core/test-files/solr
          [junit4:junit4]   2> 436 T297 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/index/src/branch_4x/solr/build/solr-core/test-files/solr/'
          [junit4:junit4]   2> 442 T297 oasc.CoreContainer.<init> New CoreContainer 1897850152
          [junit4:junit4]   2> 442 T297 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /index/src/branch_4x/solr/build/solr-core/test-files/solr/collection1/, dataDir=/index/src/branch_4x/solr/build/solr-core/test/J0/./solrtest-MBeansHandlerTest-1351102494087/
          [junit4:junit4]   2> 443 T297 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@201a970
          [junit4:junit4]   2> 443 T297 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/index/src/branch_4x/solr/build/solr-core/test/J0/./solrtest-MBeansHandlerTest-1351102494087/index/
          [junit4:junit4]   2> 444 T297 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory '/index/src/branch_4x/solr/build/solr-core/test/J0/./solrtest-MBeansHandlerTest-1351102494087/index' doesn't exist. Creating new index...
          [junit4:junit4]   2> 460 T297 oasc.CachingDirectoryFactory.get return new directory for /index/src/branch_4x/solr/build/solr-core/test/J0/./solrtest-MBeansHandlerTest-1351102494087/index forceNew:false
          [junit4:junit4]   2> 462 T297 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
          [junit4:junit4]   2>            commit{dir=MockDirWrapper(org.apache.lucene.store.SimpleFSDirectory@/index/src/branch_4x/solr/build/solr-core/test/J0/index4103741039tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@8f7fddb),segFN=segments_1,generation=1,filenames=[segments_1]
          [junit4:junit4]   2> 462 T297 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
          [junit4:junit4]   2> 462 T297 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
          [junit4:junit4]   2> 464 T297 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
          [junit4:junit4]   2> 464 T297 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
          [junit4:junit4]   2> 465 T297 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
          [junit4:junit4]   2> 465 T297 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
          [junit4:junit4]   2> 466 T297 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
          [junit4:junit4]   2> 466 T297 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
          [junit4:junit4]   2> 466 T297 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
          [junit4:junit4]   2> 467 T297 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
          [junit4:junit4]   2> 467 T297 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
          [junit4:junit4]   2> 468 T297 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
          [junit4:junit4]   2> 469 T297 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
          [junit4:junit4]   2> 469 T297 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
          [junit4:junit4]   2> 469 T297 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
          [junit4:junit4]   2> 470 T297 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
          [junit4:junit4]   2> 470 T297 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
          [junit4:junit4]   2> 471 T297 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
          [junit4:junit4]   2> 471 T297 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
          [junit4:junit4]   2> 472 T297 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
          [junit4:junit4]   2> 472 T297 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
          [junit4:junit4]   2> 473 T297 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
          [junit4:junit4]   2> 473 T297 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
          [junit4:junit4]   2> 474 T297 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
          [junit4:junit4]   2> 475 T297 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
          [junit4:junit4]   2> 475 T297 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
          [junit4:junit4]   2> 476 T297 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
          [junit4:junit4]   2> 478 T297 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
          [junit4:junit4]   2> 479 T297 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
          [junit4:junit4]   2> 481 T297 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [THROW.ERROR.ON.ADD.UPDATEPROCESSOR.JS, SOLRCONFIG-HIGHLIGHT.XML, SCHEMA-REQUIRED-FIELDS.XML, SCHEMA-MINIMAL.XML, BAD-SCHEMA-DUP-DYNAMICFIELD.XML, SCHEMA-REPLICATION2.XML, TRIVIAL.UPDATEPROCESSOR1.JS, SOLRCONFIG-CACHING.XML, SOLRCONFIG-REPEATER.XML, BAD-SCHEMA-NONTEXT-ANALYZER.XML, CURRENCY.XML, SOLRCONFIG-MERGEPOLICY.XML, SOLRCONFIG-TLOG.XML, SOLRCONFIG-MASTER.XML, BAD-SCHEMA-UNIQUEKEY-MULTIVALUED.XML, SCHEMA11.XML, SOLRCONFIG-BASIC.XML, DA_COMPOUNDDICTIONARY.TXT, MISSLEADING.EXTENSION.UPDATEPROCESSOR.JS.TXT, SCHEMA-COPYFIELD-TEST.XML, SOLRCONFIG-SLAVE.XML, SOLRCONFIG-PROPINJECT-INDEXDEFAULT.XML, ELEVATE.XML, TRIVIAL.UPDATEPROCESSOR0.JS, BAD-SOLRCONFIG-BOGUS-SCRIPTENGINE-NAME.XML, SCHEMA-CHARFILTERS.XML, SCHEMA-IB.XML, SOLRCONFIG-QUERYSENDER.XML, SCHEMA-REPLICATION1.XML, DA_UTF8.XML, SOLRCONFIG-SNIPPET-PROCESSOR.XML, CONDITIONAL.UPDATEPROCESSOR.JS, MISSING.FUNCTIONS.UPDATEPROCESSOR.JS, SOLRCONFIG-ENABLEPLUGIN.XML, HYPHENATION.DTD, SCHEMA-PHRASESUGGEST.XML, STEMDICT.TXT, HUNSPELL-TEST.AFF, SCHEMA-SNIPPET-TYPE.XML, STOPTYPES-1.TXT, STOPWORDSWRONGENCODING.TXT, SCHEMA-NUMERIC.XML, SOLRCONFIG-TRANSFORMERS.XML, SOLRCONFIG-PROPINJECT.XML, BAD-SCHEMA-NOT-INDEXED-BUT-TF.XML, SOLRCONFIG-SIMPLELOCK.XML, WDFTYPES.TXT, STOPTYPES-2.TXT, SCHEMA-REVERSED.XML, BAD-SCHEMA-CURRENCY-FT-MULTIVALUED.XML, SOLRCONFIG-SPELLCHECKCOMPONENT.XML, BAD-SCHEMA-CURRENCY-MULTIVALUED.XML, SCHEMA-DFR.XML, SOLRCONFIG-PHRASESUGGEST.XML, BAD-SCHEMA-NOT-INDEXED-BUT-POS.XML, KEEP-1.TXT, OPEN-EXCHANGE-RATES.JSON, SCHEMA-SPATIAL.XML, STOPWITHBOM.TXT, SOLRCONFIG-SPELLCHECKER.XML, SCHEMA-BINARYFIELD.XML, SOLRCONFIG-UPDATE-PROCESSOR-CHAINS.XML, BAD-SCHEMA-OMIT-TF-BUT-NOT-POS.XML, BAD-SCHEMA-DUP-FIELDTYPE.XML, SCHEMA-XINCLUDE.XML, SOLRCONFIG-MASTER1.XML, SYNONYMS.TXT, BAD-SCHEMA-CURRENCY-DYNAMIC-MULTIVALUED.XML, SCHEMA.XML, SCHEMA_CODEC.XML, SOLRCONFIG-SOLR-749.XML, SOLRCONFIG-MASTER1-KEEPONEBACKUP.XML, STOP-2.TXT, SOLRCONFIG-FUNCTIONQUERY.XML, SOLRCONFIG-TERMINDEX.XML, SCHEMA-LMDIRICHLET.XML, SOLRCONFIG-ELEVATE.XML, STOPWORDS.TXT, SCHEMA-FOLDING.XML, SCHEMA-STOP-KEEP.XML, BAD-SCHEMA-NOT-INDEXED-BUT-NORMS.XML, SOLRCONFIG-SOLCOREPROPERTIES.XML, STOP-1.TXT, SOLRCONFIG-MASTER2.XML, SCHEMA-SPELLCHECKER.XML, SOLRCONFIG-RESPONSE-LOG-COMPONENT.XML, SOLRCONFIG-LAZYWRITER.XML, SCHEMA-LUCENEMATCHVERSION.XML, FRENCHARTICLES.TXT, BAD-MP-SOLRCONFIG.XML, SCHEMA15.XML, SOLRCONFIG-REQHANDLER.INCL, SCHEMASURROUND.XML, SCHEMA-COLLATEFILTER.XML, SOLRCONFIG-MASTER3.XML, HUNSPELL-TEST.DIC, SOLRCONFIG-XINCLUDE.XML, BAD-SCHEMA-CODEC-GLOBAL-VS-FT-MISMATCH.XML, SOLRCONFIG-SLAVE1.XML, SOLRCONFIG-DELPOLICY1.XML, SCHEMA-SIM.XML, SCHEMA-SNIPPET-FIELD.XML, SCHEMA-COLLATE.XML, STOP-SNOWBALL.TXT, BAD-SCHEMA-SIM-GLOBAL-VS-FT-MISMATCH.XML, PROTWORDS.TXT, SCHEMA-TRIE.XML, SOLRCONFIG_CODEC.XML, BAD-SOLRCONFIG-INVALID-SCRIPTFILE.XML, JASUGGEST.TXT, SCHEMA-TFIDF.XML, SOLRCONFIG-SCRIPT-UPDATEPROCESSOR.XML, SCHEMA-LMJELINEKMERCER.XML, PHRASESUGGEST.TXT, BAD-SOLRCONFIG-MISSING-SCRIPTFILE.XML, SOLRCONFIG-BASIC-LUCENEVERSION31.XML, BAD-SCHEMA-UNIQUEKEY-USES-DEFAULT.XML, OLD_SYNONYMS.TXT, SOLRCONFIG-DELPOLICY2.XML, SOLRCONFIG-NATIVELOCK.XML, XSLT, BAD-SCHEMA-DUP-FIELD.XML, SOLRCONFIG-NOCACHE.XML, SCHEMA-BM25.XML, ADDFIELDS.UPDATEPROCESSOR.JS, SOLRCONFIG-QUERYSENDER-NOQUERY.XML, SOLRCONFIG-ALTDIRECTORY.XML, COMPOUNDDICTIONARY.TXT, SOLRCONFIG-INDEXCONFIG.XML, SOLRCONFIG_PERF.XML, SCHEMA-NOT-REQUIRED-UNIQUE-KEY.XML, BAD-SCHEMA-ANALYZER-CLASS-AND-NESTED.XML, KEEP-2.TXT, BAD-SCHEMA-UNIQUEKEY-IS-COPYFIELD-DEST.XML, SCHEMA12.XML, MAPPING-ISOLATIN1ACCENT.TXT, BAD_SOLRCONFIG.XML, BAD-SCHEMA-EXTERNAL-FILEFIELD.XML]
          [junit4:junit4]   2> 511 T297 oass.SolrIndexSearcher.<init> Opening Searcher@63012d7c main
          [junit4:junit4]   2> 511 T297 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
          [junit4:junit4]   2> 511 T297 oasu.CommitTracker.<init> Hard AutoCommit: disabled
          [junit4:junit4]   2> 512 T297 oasu.CommitTracker.<init> Soft AutoCommit: disabled
          [junit4:junit4]   2> 512 T297 oashc.SpellCheckComponent.inform Initializing spell checkers
          [junit4:junit4]   2> 621 T297 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
          [junit4:junit4]   2> 1097 T297 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
          [junit4:junit4]   2> 1098 T297 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
          [junit4:junit4]   2> 1098 T297 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
          [junit4:junit4]   2> 1098 T297 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
          [junit4:junit4]   2> 1099 T297 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
          [junit4:junit4]   2> 1099 T297 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
          [junit4:junit4]   2> 1099 T297 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
          [junit4:junit4]   2> 1099 T297 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
          [junit4:junit4]   2> 1100 T297 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
          [junit4:junit4]   2> 1100 T297 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
          [junit4:junit4]   2> 1108 T298 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@63012d7c main{StandardDirectoryReader(segments_1:1)}
          [junit4:junit4]   2> 1115 T297 oasc.CoreContainer.register registering core: collection1
          [junit4:junit4]   2> 1117 T297 oas.SolrTestCaseJ4.initCore ####initCore end
          [junit4:junit4]   2> 1120 T297 oas.SolrTestCaseJ4.setUp ###Starting testDiff
          [junit4:junit4]   2> ASYNC  NEW_CORE C21 name=collection1 org.apache.solr.core.SolrCore@2356df1d
          [junit4:junit4]   2> 1124 T297 C21 REQ [collection1] webapp=null path=null params={stats=true&wt=xml&qt=/admin/mbeans} status=0 QTime=3
          [junit4:junit4]   2> 1314 T297 C21 REQ [collection1] webapp=null path=null params={diff=true&stats=true&wt=xml&qt=/admin/mbeans} status=0 QTime=185
          [junit4:junit4]   2> 1342 T297 oas.SolrTestCaseJ4.tearDown ###Ending testDiff
          [junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=MBeansHandlerTest -Dtests.method=testDiff -Dtests.seed=E7FE1832E8E65C65 -Dtests.slow=true -Dtests.locale=zh_CN -Dtests.timezone=America/Indiana/Petersburg -Dtests.file.encoding=UTF-8
          [junit4:junit4] FAILURE 0.26s J0 | MBeansHandlerTest.testDiff <<<
          [junit4:junit4]    > Throwable #1: org.junit.ComparisonFailure: expected:<Was: [1, Now: 2], Delta: 1> but was:<Was: [278, Now: 279], Delta: 1>
          [junit4:junit4]    >    at __randomizedtesting.SeedInfo.seed([E7FE1832E8E65C65:22E8DCA9F8506405]:0)
          [junit4:junit4]    >    at org.junit.Assert.assertEquals(Assert.java:125)
          [junit4:junit4]    >    at org.junit.Assert.assertEquals(Assert.java:147)
          [junit4:junit4]    >    at org.apache.solr.handler.admin.MBeansHandlerTest.testDiff(MBeansHandlerTest.java:63)
          [junit4:junit4]    >    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          [junit4:junit4]    >    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          [junit4:junit4]    >    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          [junit4:junit4]    >    at java.lang.reflect.Method.invoke(Method.java:601)
          [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
          [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
          [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
          [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
          [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
          [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
          [junit4:junit4]    >    at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
          [junit4:junit4]    >    at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
          [junit4:junit4]    >    at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
          [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
          [junit4:junit4]    >    at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
          [junit4:junit4]    >    at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
          [junit4:junit4]    >    at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
          [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
          [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
          [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
          [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
          [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
          [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
          [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
          [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
          [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
          [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
          [junit4:junit4]    >    at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
          [junit4:junit4]    >    at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
          [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
          [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
          [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
          [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
          [junit4:junit4]    >    at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
          [junit4:junit4]    >    at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
          [junit4:junit4]    >    at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
          [junit4:junit4]    >    at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
          [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
          [junit4:junit4]    >    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
          [junit4:junit4]    >    at java.lang.Thread.run(Thread.java:722)
          [junit4:junit4]   2> 1375 T297 oas.SolrTestCaseJ4.deleteCore ###deleteCore
          [junit4:junit4]   2> 1376 T297 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1897850152
          [junit4:junit4]   2> 1376 T297 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@2356df1d
          [junit4:junit4]   2> 1391 T297 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
          [junit4:junit4]   2> 1392 T297 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState
          [junit4:junit4]   2> 1392 T297 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
          [junit4:junit4]   2> 1392 T297 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
          [junit4:junit4]   2> 1393 T297 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
          [junit4:junit4]   2> NOTE: test params are: codec=Lucene41: {}, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=zh_CN, timezone=America/Indiana/Petersburg
          [junit4:junit4]   2> NOTE: Linux 2.6.32-279.9.1.el6.centos.plus.x86_64 amd64/Oracle Corporation 1.7.0_09 (64-bit)/cpus=4,threads=1,free=125313208,total=262733824
          [junit4:junit4]   2> NOTE: All tests run in this JVM: [TestSolrDeletionPolicy1, JsonLoaderTest, TestArbitraryIndexDir, SolrCmdDistributorTest, ShowFileRequestHandlerTest, TestMultiCoreConfBootstrap, TestCSVResponseWriter, NotRequiredUniqueKeyTest, AlternateDirectoryTest, SearchHandlerTest, DateMathParserTest, NoCacheHeaderTest, QueryEqualityTest, OpenExchangeRatesOrgProviderTest, QueryElevationComponentTest, TestSolrDeletionPolicy2, FileUtilsTest, DirectSolrSpellCheckerTest, TestJmxMonitoredMap, TestStressLucene, TimeZoneUtilsTest, MBeansHandlerTest]
          [junit4:junit4] Completed on J0 in 1.41s, 1 test, 1 failure <<< FAILURES!
          
          Show
          Shawn Heisey added a comment - - edited With this as the contructor: public RequestHandlerBase() { // numRequests = Metrics.newCounter(RequestHandlerBase.class, "numRequests" , this .toString()); // numErrors = Metrics.newCounter(RequestHandlerBase.class, "numErrors" , this .toString()); // numTimeouts = Metrics.newCounter(RequestHandlerBase.class, "numTimeouts" , this .toString()); // requestTimes = Metrics.newTimer(RequestHandlerBase.class, "requestTimes" , this .toString()); numRequests = Metrics.newCounter(RequestHandlerBase.class, "numRequests" ); numErrors = Metrics.newCounter(RequestHandlerBase.class, "numErrors" ); numTimeouts = Metrics.newCounter(RequestHandlerBase.class, "numTimeouts" ); requestTimes = Metrics.newTimer(RequestHandlerBase.class, "requestTimes" ); } I get the following as the failure. I suppose I should be glad that it fails, but RequestHandlersTests, which is the test that was modified, continues to pass. [junit4:junit4] Suite: org.apache.solr.handler.admin.MBeansHandlerTest [junit4:junit4] 2> 0 T297 oas.SolrTestCaseJ4.initCore ####initCore [junit4:junit4] 2> Creating dataDir: /index/src/branch_4x/solr/build/solr-core/test/J0/./solrtest-MBeansHandlerTest-1351102494087 [junit4:junit4] 2> 26 T297 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/index/src/branch_4x/solr/build/solr-core/test-files/solr/collection1/' [junit4:junit4] 2> 27 T297 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/index/src/branch_4x/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader [junit4:junit4] 2> 27 T297 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/index/src/branch_4x/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader [junit4:junit4] 2> 59 T297 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41 [junit4:junit4] 2> 98 T297 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml [junit4:junit4] 2> 98 T297 oass.IndexSchema.readSchema Reading Solr Schema [junit4:junit4] 2> 105 T297 oass.IndexSchema.readSchema Schema name=test [junit4:junit4] 2> 414 T297 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440. [junit4:junit4] 2> 420 T297 oass.IndexSchema.readSchema default search field in schema is text [junit4:junit4] 2> 422 T297 oass.IndexSchema.readSchema unique key field: id [junit4:junit4] 2> 430 T297 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml [junit4:junit4] 2> 432 T297 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml [junit4:junit4] 2> 434 T297 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json [junit4:junit4] 2> 435 T297 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json [junit4:junit4] 2> 436 T297 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx) [junit4:junit4] 2> 436 T297 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /index/src/branch_4x/solr/build/solr-core/test-files/solr [junit4:junit4] 2> 436 T297 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/index/src/branch_4x/solr/build/solr-core/test-files/solr/' [junit4:junit4] 2> 442 T297 oasc.CoreContainer.<init> New CoreContainer 1897850152 [junit4:junit4] 2> 442 T297 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /index/src/branch_4x/solr/build/solr-core/test-files/solr/collection1/, dataDir=/index/src/branch_4x/solr/build/solr-core/test/J0/./solrtest-MBeansHandlerTest-1351102494087/ [junit4:junit4] 2> 443 T297 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@201a970 [junit4:junit4] 2> 443 T297 oasc.SolrCore.getNewIndexDir New index directory detected: old= null new =/index/src/branch_4x/solr/build/solr-core/test/J0/./solrtest-MBeansHandlerTest-1351102494087/index/ [junit4:junit4] 2> 444 T297 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory '/index/src/branch_4x/solr/build/solr-core/test/J0/./solrtest-MBeansHandlerTest-1351102494087/index' doesn't exist. Creating new index... [junit4:junit4] 2> 460 T297 oasc.CachingDirectoryFactory.get return new directory for /index/src/branch_4x/solr/build/solr-core/test/J0/./solrtest-MBeansHandlerTest-1351102494087/index forceNew: false [junit4:junit4] 2> 462 T297 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1 [junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.SimpleFSDirectory@/index/src/branch_4x/solr/build/solr-core/test/J0/index4103741039tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@8f7fddb),segFN=segments_1,generation=1,filenames=[segments_1] [junit4:junit4] 2> 462 T297 oasc.SolrDeletionPolicy.updateCommits newest commit = 1 [junit4:junit4] 2> 462 T297 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter [junit4:junit4] 2> 464 T297 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe" [junit4:junit4] 2> 464 T297 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields" [junit4:junit4] 2> 465 T297 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig" [junit4:junit4] 2> 465 T297 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields" [junit4:junit4] 2> 466 T297 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit" [junit4:junit4] 2> 466 T297 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler [junit4:junit4] 2> 466 T297 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler [junit4:junit4] 2> 467 T297 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler [junit4:junit4] 2> 467 T297 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler [junit4:junit4] 2> 468 T297 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler [junit4:junit4] 2> 469 T297 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler [junit4:junit4] 2> 469 T297 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers [junit4:junit4] 2> 469 T297 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler [junit4:junit4] 2> 470 T297 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler [junit4:junit4] 2> 470 T297 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler [junit4:junit4] 2> 471 T297 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler [junit4:junit4] 2> 471 T297 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler [junit4:junit4] 2> 472 T297 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler [junit4:junit4] 2> 472 T297 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler [junit4:junit4] 2> 473 T297 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler [junit4:junit4] 2> 473 T297 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler [junit4:junit4] 2> 474 T297 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler [junit4:junit4] 2> 475 T297 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler [junit4:junit4] 2> 475 T297 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler [junit4:junit4] 2> 476 T297 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler [junit4:junit4] 2> 478 T297 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60 [junit4:junit4] 2> 479 T297 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler. [junit4:junit4] 2> 481 T297 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [THROW.ERROR.ON.ADD.UPDATEPROCESSOR.JS, SOLRCONFIG-HIGHLIGHT.XML, SCHEMA-REQUIRED-FIELDS.XML, SCHEMA-MINIMAL.XML, BAD-SCHEMA-DUP-DYNAMICFIELD.XML, SCHEMA-REPLICATION2.XML, TRIVIAL.UPDATEPROCESSOR1.JS, SOLRCONFIG-CACHING.XML, SOLRCONFIG-REPEATER.XML, BAD-SCHEMA-NONTEXT-ANALYZER.XML, CURRENCY.XML, SOLRCONFIG-MERGEPOLICY.XML, SOLRCONFIG-TLOG.XML, SOLRCONFIG-MASTER.XML, BAD-SCHEMA-UNIQUEKEY-MULTIVALUED.XML, SCHEMA11.XML, SOLRCONFIG-BASIC.XML, DA_COMPOUNDDICTIONARY.TXT, MISSLEADING.EXTENSION.UPDATEPROCESSOR.JS.TXT, SCHEMA-COPYFIELD-TEST.XML, SOLRCONFIG-SLAVE.XML, SOLRCONFIG-PROPINJECT-INDEXDEFAULT.XML, ELEVATE.XML, TRIVIAL.UPDATEPROCESSOR0.JS, BAD-SOLRCONFIG-BOGUS-SCRIPTENGINE-NAME.XML, SCHEMA-CHARFILTERS.XML, SCHEMA-IB.XML, SOLRCONFIG-QUERYSENDER.XML, SCHEMA-REPLICATION1.XML, DA_UTF8.XML, SOLRCONFIG-SNIPPET-PROCESSOR.XML, CONDITIONAL.UPDATEPROCESSOR.JS, MISSING.FUNCTIONS.UPDATEPROCESSOR.JS, SOLRCONFIG-ENABLEPLUGIN.XML, HYPHENATION.DTD, SCHEMA-PHRASESUGGEST.XML, STEMDICT.TXT, HUNSPELL-TEST.AFF, SCHEMA-SNIPPET-TYPE.XML, STOPTYPES-1.TXT, STOPWORDSWRONGENCODING.TXT, SCHEMA-NUMERIC.XML, SOLRCONFIG-TRANSFORMERS.XML, SOLRCONFIG-PROPINJECT.XML, BAD-SCHEMA-NOT-INDEXED-BUT-TF.XML, SOLRCONFIG-SIMPLELOCK.XML, WDFTYPES.TXT, STOPTYPES-2.TXT, SCHEMA-REVERSED.XML, BAD-SCHEMA-CURRENCY-FT-MULTIVALUED.XML, SOLRCONFIG-SPELLCHECKCOMPONENT.XML, BAD-SCHEMA-CURRENCY-MULTIVALUED.XML, SCHEMA-DFR.XML, SOLRCONFIG-PHRASESUGGEST.XML, BAD-SCHEMA-NOT-INDEXED-BUT-POS.XML, KEEP-1.TXT, OPEN-EXCHANGE-RATES.JSON, SCHEMA-SPATIAL.XML, STOPWITHBOM.TXT, SOLRCONFIG-SPELLCHECKER.XML, SCHEMA-BINARYFIELD.XML, SOLRCONFIG-UPDATE-PROCESSOR-CHAINS.XML, BAD-SCHEMA-OMIT-TF-BUT-NOT-POS.XML, BAD-SCHEMA-DUP-FIELDTYPE.XML, SCHEMA-XINCLUDE.XML, SOLRCONFIG-MASTER1.XML, SYNONYMS.TXT, BAD-SCHEMA-CURRENCY-DYNAMIC-MULTIVALUED.XML, SCHEMA.XML, SCHEMA_CODEC.XML, SOLRCONFIG-SOLR-749.XML, SOLRCONFIG-MASTER1-KEEPONEBACKUP.XML, STOP-2.TXT, SOLRCONFIG-FUNCTIONQUERY.XML, SOLRCONFIG-TERMINDEX.XML, SCHEMA-LMDIRICHLET.XML, SOLRCONFIG-ELEVATE.XML, STOPWORDS.TXT, SCHEMA-FOLDING.XML, SCHEMA-STOP-KEEP.XML, BAD-SCHEMA-NOT-INDEXED-BUT-NORMS.XML, SOLRCONFIG-SOLCOREPROPERTIES.XML, STOP-1.TXT, SOLRCONFIG-MASTER2.XML, SCHEMA-SPELLCHECKER.XML, SOLRCONFIG-RESPONSE-LOG-COMPONENT.XML, SOLRCONFIG-LAZYWRITER.XML, SCHEMA-LUCENEMATCHVERSION.XML, FRENCHARTICLES.TXT, BAD-MP-SOLRCONFIG.XML, SCHEMA15.XML, SOLRCONFIG-REQHANDLER.INCL, SCHEMASURROUND.XML, SCHEMA-COLLATEFILTER.XML, SOLRCONFIG-MASTER3.XML, HUNSPELL-TEST.DIC, SOLRCONFIG-XINCLUDE.XML, BAD-SCHEMA-CODEC-GLOBAL-VS-FT-MISMATCH.XML, SOLRCONFIG-SLAVE1.XML, SOLRCONFIG-DELPOLICY1.XML, SCHEMA-SIM.XML, SCHEMA-SNIPPET-FIELD.XML, SCHEMA-COLLATE.XML, STOP-SNOWBALL.TXT, BAD-SCHEMA-SIM-GLOBAL-VS-FT-MISMATCH.XML, PROTWORDS.TXT, SCHEMA-TRIE.XML, SOLRCONFIG_CODEC.XML, BAD-SOLRCONFIG-INVALID-SCRIPTFILE.XML, JASUGGEST.TXT, SCHEMA-TFIDF.XML, SOLRCONFIG-SCRIPT-UPDATEPROCESSOR.XML, SCHEMA-LMJELINEKMERCER.XML, PHRASESUGGEST.TXT, BAD-SOLRCONFIG-MISSING-SCRIPTFILE.XML, SOLRCONFIG-BASIC-LUCENEVERSION31.XML, BAD-SCHEMA-UNIQUEKEY-USES-DEFAULT.XML, OLD_SYNONYMS.TXT, SOLRCONFIG-DELPOLICY2.XML, SOLRCONFIG-NATIVELOCK.XML, XSLT, BAD-SCHEMA-DUP-FIELD.XML, SOLRCONFIG-NOCACHE.XML, SCHEMA-BM25.XML, ADDFIELDS.UPDATEPROCESSOR.JS, SOLRCONFIG-QUERYSENDER-NOQUERY.XML, SOLRCONFIG-ALTDIRECTORY.XML, COMPOUNDDICTIONARY.TXT, SOLRCONFIG-INDEXCONFIG.XML, SOLRCONFIG_PERF.XML, SCHEMA-NOT-REQUIRED-UNIQUE-KEY.XML, BAD-SCHEMA-ANALYZER-CLASS-AND-NESTED.XML, KEEP-2.TXT, BAD-SCHEMA-UNIQUEKEY-IS-COPYFIELD-DEST.XML, SCHEMA12.XML, MAPPING-ISOLATIN1ACCENT.TXT, BAD_SOLRCONFIG.XML, BAD-SCHEMA-EXTERNAL-FILEFIELD.XML] [junit4:junit4] 2> 511 T297 oass.SolrIndexSearcher.<init> Opening Searcher@63012d7c main [junit4:junit4] 2> 511 T297 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper [junit4:junit4] 2> 511 T297 oasu.CommitTracker.<init> Hard AutoCommit: disabled [junit4:junit4] 2> 512 T297 oasu.CommitTracker.<init> Soft AutoCommit: disabled [junit4:junit4] 2> 512 T297 oashc.SpellCheckComponent.inform Initializing spell checkers [junit4:junit4] 2> 621 T297 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3} [junit4:junit4] 2> 1097 T297 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0 [junit4:junit4] 2> 1098 T297 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http: // [junit4:junit4] 2> 1098 T297 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0 [junit4:junit4] 2> 1098 T297 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20 [junit4:junit4] 2> 1099 T297 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0 [junit4:junit4] 2> 1099 T297 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647 [junit4:junit4] 2> 1099 T297 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5 [junit4:junit4] 2> 1099 T297 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1 [junit4:junit4] 2> 1100 T297 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false [junit4:junit4] 2> 1100 T297 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry= false [junit4:junit4] 2> 1108 T298 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@63012d7c main{StandardDirectoryReader(segments_1:1)} [junit4:junit4] 2> 1115 T297 oasc.CoreContainer.register registering core: collection1 [junit4:junit4] 2> 1117 T297 oas.SolrTestCaseJ4.initCore ####initCore end [junit4:junit4] 2> 1120 T297 oas.SolrTestCaseJ4.setUp ###Starting testDiff [junit4:junit4] 2> ASYNC NEW_CORE C21 name=collection1 org.apache.solr.core.SolrCore@2356df1d [junit4:junit4] 2> 1124 T297 C21 REQ [collection1] webapp= null path= null params={stats= true &wt=xml&qt=/admin/mbeans} status=0 QTime=3 [junit4:junit4] 2> 1314 T297 C21 REQ [collection1] webapp= null path= null params={diff= true &stats= true &wt=xml&qt=/admin/mbeans} status=0 QTime=185 [junit4:junit4] 2> 1342 T297 oas.SolrTestCaseJ4.tearDown ###Ending testDiff [junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=MBeansHandlerTest -Dtests.method=testDiff -Dtests.seed=E7FE1832E8E65C65 -Dtests.slow= true -Dtests.locale=zh_CN -Dtests.timezone=America/Indiana/Petersburg -Dtests.file.encoding=UTF-8 [junit4:junit4] FAILURE 0.26s J0 | MBeansHandlerTest.testDiff <<< [junit4:junit4] > Throwable #1: org.junit.ComparisonFailure: expected:<Was: [1, Now: 2], Delta: 1> but was:<Was: [278, Now: 279], Delta: 1> [junit4:junit4] > at __randomizedtesting.SeedInfo.seed([E7FE1832E8E65C65:22E8DCA9F8506405]:0) [junit4:junit4] > at org.junit.Assert.assertEquals(Assert.java:125) [junit4:junit4] > at org.junit.Assert.assertEquals(Assert.java:147) [junit4:junit4] > at org.apache.solr.handler.admin.MBeansHandlerTest.testDiff(MBeansHandlerTest.java:63) [junit4:junit4] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [junit4:junit4] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [junit4:junit4] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [junit4:junit4] > at java.lang.reflect.Method.invoke(Method.java:601) [junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559) [junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79) [junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737) [junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773) [junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787) [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) [junit4:junit4] > at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) [junit4:junit4] > at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51) [junit4:junit4] > at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) [junit4:junit4] > at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) [junit4:junit4] > at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70) [junit4:junit4] > at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358) [junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782) [junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442) [junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746) [junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648) [junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682) [junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693) [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) [junit4:junit4] > at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [junit4:junit4] > at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4:junit4] > at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43) [junit4:junit4] > at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) [junit4:junit4] > at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70) [junit4:junit4] > at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358) [junit4:junit4] > at java.lang. Thread .run( Thread .java:722) [junit4:junit4] 2> 1375 T297 oas.SolrTestCaseJ4.deleteCore ###deleteCore [junit4:junit4] 2> 1376 T297 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1897850152 [junit4:junit4] 2> 1376 T297 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@2356df1d [junit4:junit4] 2> 1391 T297 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0} [junit4:junit4] 2> 1392 T297 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState [junit4:junit4] 2> 1392 T297 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter [junit4:junit4] 2> 1392 T297 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser [junit4:junit4] 2> 1393 T297 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request. [junit4:junit4] 2> NOTE: test params are: codec=Lucene41: {}, sim=RandomSimilarityProvider(queryNorm= true ,coord=no): {}, locale=zh_CN, timezone=America/Indiana/Petersburg [junit4:junit4] 2> NOTE: Linux 2.6.32-279.9.1.el6.centos.plus.x86_64 amd64/Oracle Corporation 1.7.0_09 (64-bit)/cpus=4,threads=1,free=125313208,total=262733824 [junit4:junit4] 2> NOTE: All tests run in this JVM: [TestSolrDeletionPolicy1, JsonLoaderTest, TestArbitraryIndexDir, SolrCmdDistributorTest, ShowFileRequestHandlerTest, TestMultiCoreConfBootstrap, TestCSVResponseWriter, NotRequiredUniqueKeyTest, AlternateDirectoryTest, SearchHandlerTest, DateMathParserTest, NoCacheHeaderTest, QueryEqualityTest, OpenExchangeRatesOrgProviderTest, QueryElevationComponentTest, TestSolrDeletionPolicy2, FileUtilsTest, DirectSolrSpellCheckerTest, TestJmxMonitoredMap, TestStressLucene, TimeZoneUtilsTest, MBeansHandlerTest] [junit4:junit4] Completed on J0 in 1.41s, 1 test, 1 failure <<< FAILURES!
          Hide
          Alan Woodward added a comment -

          There's a test bug in there - it's comparing the NamedList objects when it should be comparing their values. Will put up a patch in a bit...

          Show
          Alan Woodward added a comment - There's a test bug in there - it's comparing the NamedList objects when it should be comparing their values. Will put up a patch in a bit...
          Hide
          Alan Woodward added a comment -

          Patch correcting the test bug (after an hour or so of swearing at Java object equality semantics...)

          Show
          Alan Woodward added a comment - Patch correcting the test bug (after an hour or so of swearing at Java object equality semantics...)
          Hide
          Shawn Heisey added a comment -

          You probably already knew this, but now the updated test fails properly for me when the constructor doesn't set the scope. I still get the failure in MBeansHandlerTest, which without knowledge of how all this stuff works internally, is really really mystifying. Since all solr tests pass when I don't monkey with the constructor, I guess it's not a big deal.

          Not that my vote really counts, but +1 for committing to 4x and trunk from me. Take out those enemy death cannons.

          Show
          Shawn Heisey added a comment - You probably already knew this, but now the updated test fails properly for me when the constructor doesn't set the scope. I still get the failure in MBeansHandlerTest, which without knowledge of how all this stuff works internally, is really really mystifying. Since all solr tests pass when I don't monkey with the constructor, I guess it's not a big deal. Not that my vote really counts, but +1 for committing to 4x and trunk from me. Take out those enemy death cannons.
          Hide
          Alan Woodward added a comment -

          Thanks for testing it Shawn.

          I think I'd like some more eyes on it before I commit - it adds a dependency to solr-core, which is a pretty big change. Anyone else have an opinion?

          Show
          Alan Woodward added a comment - Thanks for testing it Shawn. I think I'd like some more eyes on it before I commit - it adds a dependency to solr-core, which is a pretty big change. Anyone else have an opinion?
          Hide
          Adrien Grand added a comment -

          Hi Alan,

          This is great to have all these statistics available through JMX, this would be really helpful for Solr monitoring! Instead of relying on RequestHandlerBase.toString to return a distinct instance per request handler, maybe each request handler should have its own MetricsRegistry (or maybe use the request handler path as the scope, but this looks harder to do)?

          I have no objection to adding the metrics jar to the dependencies of solr-core.

          Show
          Adrien Grand added a comment - Hi Alan, This is great to have all these statistics available through JMX, this would be really helpful for Solr monitoring! Instead of relying on RequestHandlerBase.toString to return a distinct instance per request handler, maybe each request handler should have its own MetricsRegistry (or maybe use the request handler path as the scope, but this looks harder to do)? I have no objection to adding the metrics jar to the dependencies of solr-core.
          Hide
          Alan Woodward added a comment -

          Yeah, the toString hack is quite ... hacky. The problem with using the request handler path as the scope (which I agree would be logical) is that this isn't available until init() is called, and both getStatistics() and handleRequest() can get called before init().

          Having one MetricsRegistry per request handler looks a lot more sensible though. I'll put a patch together. Thanks!

          Show
          Alan Woodward added a comment - Yeah, the toString hack is quite ... hacky. The problem with using the request handler path as the scope (which I agree would be logical) is that this isn't available until init() is called, and both getStatistics() and handleRequest() can get called before init(). Having one MetricsRegistry per request handler looks a lot more sensible though. I'll put a patch together. Thanks!
          Hide
          Alan Woodward added a comment -

          This patch declares a MetricsRegistry in the RequestHandlerBase constructor, and uses that to ensure that the various metrics are held per-handler.

          There doesn't seem to be any way of setting the metric names after construction, so I don't think we'll be able to write the handler path into the metrics, unfortunately. But you'll be able to tell the various handlers apart by accessing the data through the existing SOLR JMX beans.

          Show
          Alan Woodward added a comment - This patch declares a MetricsRegistry in the RequestHandlerBase constructor, and uses that to ensure that the various metrics are held per-handler. There doesn't seem to be any way of setting the metric names after construction, so I don't think we'll be able to write the handler path into the metrics, unfortunately. But you'll be able to tell the various handlers apart by accessing the data through the existing SOLR JMX beans.
          Hide
          Shawn Heisey added a comment -

          I ran into some trouble with tests failing, so I checked out a fresh branch_4x, applied this patch, and tried running solr tests. There are major explosions. I'll be attaching the full putty log for review. It appears that the first exception is an out of memory error:

          [junit4:junit4] 2> 7800 T177 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[metrics-meter-tick-thread-1,5,TGRP-TestRandomFaceting] java.lang.OutOfMemoryError: unable to create new native thread
          [junit4:junit4] 2> at __randomizedtesting.SeedInfo.seed([2191B18B87EDCD66]:0)
          [junit4:junit4] 2> at java.lang.Thread.start0(Native Method)
          [junit4:junit4] 2> at java.lang.Thread.start(Thread.java:691)
          [junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943)
          [junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:992)
          [junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
          [junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
          [junit4:junit4] 2> at java.lang.Thread.run(Thread.java:722)
          [junit4:junit4] 2>

          If I edit out the MetricsRegistry and go back to creating the objects directly from Metrics and using this.toString() as the scope, the tests all pass.

          The Metrics documentation talks about MetricsRegistry as being something you create on a per-application basis. That suggests that it's a very heavy object, and even a barebones Solr install probably has at least a dozen requestHandlers defined. I don't know how many are defined in the JVMs used for testing.

          On my test branch_4x installation, I see 29 handlers between QUERYHANDLER and UPDATEHANDLER in the stats visible in the gui, and that's just on one core. I've got 16 cores defined.

          Show
          Shawn Heisey added a comment - I ran into some trouble with tests failing, so I checked out a fresh branch_4x, applied this patch, and tried running solr tests. There are major explosions. I'll be attaching the full putty log for review. It appears that the first exception is an out of memory error: [junit4:junit4] 2> 7800 T177 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread [metrics-meter-tick-thread-1,5,TGRP-TestRandomFaceting] java.lang.OutOfMemoryError: unable to create new native thread [junit4:junit4] 2> at __randomizedtesting.SeedInfo.seed( [2191B18B87EDCD66] :0) [junit4:junit4] 2> at java.lang.Thread.start0(Native Method) [junit4:junit4] 2> at java.lang.Thread.start(Thread.java:691) [junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943) [junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:992) [junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [junit4:junit4] 2> at java.lang.Thread.run(Thread.java:722) [junit4:junit4] 2> If I edit out the MetricsRegistry and go back to creating the objects directly from Metrics and using this.toString() as the scope, the tests all pass. The Metrics documentation talks about MetricsRegistry as being something you create on a per-application basis. That suggests that it's a very heavy object, and even a barebones Solr install probably has at least a dozen requestHandlers defined. I don't know how many are defined in the JVMs used for testing. On my test branch_4x installation, I see 29 handlers between QUERYHANDLER and UPDATEHANDLER in the stats visible in the gui, and that's just on one core. I've got 16 cores defined.
          Hide
          Shawn Heisey added a comment -

          Full putty log showing actions taken right after checkout of branch_4x.

          Show
          Shawn Heisey added a comment - Full putty log showing actions taken right after checkout of branch_4x.
          Hide
          Alan Woodward added a comment -

          Eek, yes, I get that as well. Must have been unlucky on my first test run.

          Looks like the previous patch is the way to go.

          Show
          Alan Woodward added a comment - Eek, yes, I get that as well. Must have been unlucky on my first test run. Looks like the previous patch is the way to go.
          Hide
          Adrien Grand added a comment -

          In that case could we use something else than the string representation as a scope? (like a randomly-generated string that is unique across RequestHandlerBase instances?)

          Show
          Adrien Grand added a comment - In that case could we use something else than the string representation as a scope? (like a randomly-generated string that is unique across RequestHandlerBase instances?)
          Hide
          Alan Woodward added a comment -

          The toString() is sort of a randomly-generated string, really. It's the default Object toString, ie a class name and a memory address.

          Show
          Alan Woodward added a comment - The toString() is sort of a randomly-generated string, really. It's the default Object toString, ie a class name and a memory address.
          Hide
          Adrien Grand added a comment -

          Can it break if people write custom request handlers that override toString?

          Show
          Adrien Grand added a comment - Can it break if people write custom request handlers that override toString?
          Hide
          Alan Woodward added a comment -

          I suppose if we had two custom request handlers that both overrode it with the same String it could break.

          How about using super.toString() instead of this.toString()? That way it's always the parent Object method that's called.

          Show
          Alan Woodward added a comment - I suppose if we had two custom request handlers that both overrode it with the same String it could break. How about using super.toString() instead of this.toString()? That way it's always the parent Object method that's called.
          Hide
          Adrien Grand added a comment -

          After reading the documentation of Object.toString http://docs.oracle.com/javase/6/docs/api/java/lang/Object.html#toString() it seems to me that it can still break if the hashCode is overriden with an impl that is likely to return the same value for 2 different instances. I know this is very unlikely but on the other hand the bug would be very hard to track... so I think we should rather generate unique strings (unique a static atomic counter for example) to make sure timers and counters are never shared?

          Show
          Adrien Grand added a comment - After reading the documentation of Object.toString http://docs.oracle.com/javase/6/docs/api/java/lang/Object.html#toString( ) it seems to me that it can still break if the hashCode is overriden with an impl that is likely to return the same value for 2 different instances. I know this is very unlikely but on the other hand the bug would be very hard to track... so I think we should rather generate unique strings (unique a static atomic counter for example) to make sure timers and counters are never shared?
          Hide
          Alan Woodward added a comment -

          In this patch, the scope name is generated from a static AtomicLong (which is basically what Shawn suggested a few days ago before I tried to be clever .

          All tests pass

          Show
          Alan Woodward added a comment - In this patch, the scope name is generated from a static AtomicLong (which is basically what Shawn suggested a few days ago before I tried to be clever . All tests pass
          Hide
          Adrien Grand added a comment -

          +1

          Show
          Adrien Grand added a comment - +1
          Hide
          Lance Norskog added a comment -

          The 5th percentile is really useful. There is always a maximum query time of 30s just because of a garbage collection failure, and people look at that number and freak out. For query times, the 5th percentile shows what is repeatedly "too slow".

          Show
          Lance Norskog added a comment - The 5th percentile is really useful. There is always a maximum query time of 30s just because of a garbage collection failure, and people look at that number and freak out. For query times, the 5th percentile shows what is repeatedly "too slow".
          Hide
          Alan Woodward added a comment -

          Hi Lance, in this case, that's represented by the 95th percentile - lower numbers mean faster query times.

          I'll commit this later this evening if no-one objects.

          Show
          Alan Woodward added a comment - Hi Lance, in this case, that's represented by the 95th percentile - lower numbers mean faster query times. I'll commit this later this evening if no-one objects.
          Hide
          Alan Woodward added a comment -

          Committed to trunk, r1403555. I'll let it sit here for a bit before back-porting.

          Show
          Alan Woodward added a comment - Committed to trunk, r1403555. I'll let it sit here for a bit before back-porting.
          Hide
          Shawn Heisey added a comment -

          Now that metrics is in trunk, there are a zillion potential uses for it for improving visibility into Solr performance. Would it be better to have one big Jira for adding these (or for depending on additional issues) or would it be better to just go ahead and create an individual issue each time I think of one? At this moment, I only have one candidate, but I'm sure I'll think of more later.

          Show
          Shawn Heisey added a comment - Now that metrics is in trunk, there are a zillion potential uses for it for improving visibility into Solr performance. Would it be better to have one big Jira for adding these (or for depending on additional issues) or would it be better to just go ahead and create an individual issue each time I think of one? At this moment, I only have one candidate, but I'm sure I'll think of more later.
          Hide
          Adrien Grand added a comment -

          Hi Shawn, I personally prefer small issues. They make it easier to get stuff committed.

          Show
          Adrien Grand added a comment - Hi Shawn, I personally prefer small issues. They make it easier to get stuff committed.
          Hide
          Shawn Heisey added a comment -

          The r1413047 commit caused a minor hiccup for this patch against branch_4x. Attached updated patch. Double-checked that it works against a clean branch_4x checkout and that all solr tests pass.

          Show
          Shawn Heisey added a comment - The r1413047 commit caused a minor hiccup for this patch against branch_4x. Attached updated patch. Double-checked that it works against a clean branch_4x checkout and that all solr tests pass.
          Hide
          Alan Woodward added a comment -

          Committed to branch 4x, r1413797

          Thanks for prodding, Shawn!

          Show
          Alan Woodward added a comment - Committed to branch 4x, r1413797 Thanks for prodding, Shawn!
          Hide
          Commit Tag Bot added a comment -

          [branch_4x commit] Alan Woodward
          http://svn.apache.org/viewvc?view=revision&revision=1413797

          SOLR-1972: Add extra query stats to RequestHandler

          Show
          Commit Tag Bot added a comment - [branch_4x commit] Alan Woodward http://svn.apache.org/viewvc?view=revision&revision=1413797 SOLR-1972 : Add extra query stats to RequestHandler
          Hide
          Mark Miller added a comment -

          Hey guys, any reason this is not on 5x? That's going to make it difficult in the future...

          Show
          Mark Miller added a comment - Hey guys, any reason this is not on 5x? That's going to make it difficult in the future...
          Hide
          Shawn Heisey added a comment -

          Hey guys, any reason this is not on 5x? That's going to make it difficult in the future...

          It looks like r1403555 added this to trunk on 2012/10/29.

          Show
          Shawn Heisey added a comment - Hey guys, any reason this is not on 5x? That's going to make it difficult in the future... It looks like r1403555 added this to trunk on 2012/10/29.
          Hide
          Alan Woodward added a comment -

          Hi Mark, I've marked this as fixed for 5.0 as well - was it just the tags that made you think it was only on 4x or is it giving you trouble merging?

          Show
          Alan Woodward added a comment - Hi Mark, I've marked this as fixed for 5.0 as well - was it just the tags that made you think it was only on 4x or is it giving you trouble merging?
          Hide
          Mark Miller added a comment -

          Yeah, I had a merge conflict in changes and didn't see the entry in 5x changes.

          Show
          Mark Miller added a comment - Yeah, I had a merge conflict in changes and didn't see the entry in 5x changes.
          Hide
          Uwe Schindler added a comment - - edited

          This commit causes a huge memory leak in Solr: The whole heap (60% while running tests) is filled with (interned) Strings, ConcurrentHashMap.Entry, and lots of class instances from the com.yammer.metrics package. This causes the the recent permgen issues in running Solr tests.

          We should revert this and investigate how to remove the com.yammer.metrics package dependency (or make the stats cleaner). To me it looks like every query to solr creates a new entry in those huge maps, causing them to grow and grow and grow... There is also no cleanup that shuts down the MBean servers holding all those references on core shutdown.

          See: http://lucene.472066.n3.nabble.com/JENKINS-Lucene-Solr-trunk-Linux-32bit-jdk1-6-0-37-Build-3421-Failure-td4029050.html

          Show
          Uwe Schindler added a comment - - edited This commit causes a huge memory leak in Solr: The whole heap (60% while running tests) is filled with (interned) Strings, ConcurrentHashMap.Entry, and lots of class instances from the com.yammer.metrics package. This causes the the recent permgen issues in running Solr tests. We should revert this and investigate how to remove the com.yammer.metrics package dependency (or make the stats cleaner). To me it looks like every query to solr creates a new entry in those huge maps, causing them to grow and grow and grow... There is also no cleanup that shuts down the MBean servers holding all those references on core shutdown. See: http://lucene.472066.n3.nabble.com/JENKINS-Lucene-Solr-trunk-Linux-32bit-jdk1-6-0-37-Build-3421-Failure-td4029050.html
          Hide
          Shawn Heisey added a comment -

          This is really useful to me, I'm very sad to learn it's causing problems. I haven't noticed anything in my own running of branch_4x, which runs for days at a time. I'm attempting to get some heap dumps so I can compare them. Based on what I know from my own experience, I don't think this is actually a leak, but even a minimal test JVM ends up with a lot of request handlers, each of which has these additional objects. As for seeing what looks like a leak with each query, after 1024 queries, the growth in non-garbage objects should stop, because it throws away an entry to make room for the new one.

          I wonder if there might be an easy way to make these new statistics optional in the request handler, so they do not cause memory problems with minimal test configs. Although it's really cool to be able to see detailed query statistics on the SolrInfoMBeanHandler, it's unnecessary.

          If it does get removed, I guess I'll have to go back to the previous version of the patch that only has additional statistics on qtime. I depend on these statistics - the average values otherwise available are nearly useless.

          I will attempt a patch. No guarantees about success!

          Show
          Shawn Heisey added a comment - This is really useful to me, I'm very sad to learn it's causing problems. I haven't noticed anything in my own running of branch_4x, which runs for days at a time. I'm attempting to get some heap dumps so I can compare them. Based on what I know from my own experience, I don't think this is actually a leak, but even a minimal test JVM ends up with a lot of request handlers, each of which has these additional objects. As for seeing what looks like a leak with each query, after 1024 queries, the growth in non-garbage objects should stop, because it throws away an entry to make room for the new one. I wonder if there might be an easy way to make these new statistics optional in the request handler, so they do not cause memory problems with minimal test configs. Although it's really cool to be able to see detailed query statistics on the SolrInfoMBeanHandler, it's unnecessary. If it does get removed, I guess I'll have to go back to the previous version of the patch that only has additional statistics on qtime. I depend on these statistics - the average values otherwise available are nearly useless. I will attempt a patch. No guarantees about success!
          Hide
          Robert Muir added a comment -

          This commit causes a huge memory leak in Solr
          ...
          We should revert this and investigate how to remove the com.yammer.metrics package dependency (or make the stats cleaner). To me it looks like every query to solr creates a new entry in those huge maps, causing them to grow and grow and grow...

          +1

          Maybe the stats have to be implemented by hand or something. I don't understand why huge maps or string interning is necessary here.

          Lets back out the change and implement it in a non-leaky way.

          Show
          Robert Muir added a comment - This commit causes a huge memory leak in Solr ... We should revert this and investigate how to remove the com.yammer.metrics package dependency (or make the stats cleaner). To me it looks like every query to solr creates a new entry in those huge maps, causing them to grow and grow and grow... +1 Maybe the stats have to be implemented by hand or something. I don't understand why huge maps or string interning is necessary here. Lets back out the change and implement it in a non-leaky way.
          Hide
          Uwe Schindler added a comment - - edited

          As for seeing what looks like a leak with each query, after 1024 queries, the growth in non-garbage objects should stop, because it throws away an entry to make room for the new one.

          That might be correct, the problem here is a leak, that might also affect people reloading Solr in their App-Server. The problem is that somehow on CoreShutdown the MBeans are not destroyed. Maybe this is fine how it is implemented, but when the Solr core shuts down, all statistics data have to be removed and nulled out or whatever.

          I would revert this patch for now, until there is a better solution. Once you have a new patch, that does not leak those objects on core shutdowns.

          The horrible thing is that all those huge maps contains stupid strings, all starting with some solr class name (a RequestHandler), followed by an id and various other information. There are (in the dump analyzed with visualvm) millions of String looking like that (OQL executed on heap dump):

          select count(heap.objects("java.lang.String"), 'it.toString().startsWith("org.apache.solr.handler")')
          

          returns: 305064 (to just show some of the string instances hanging around)

          This looks like a huge waste!

          Show
          Uwe Schindler added a comment - - edited As for seeing what looks like a leak with each query, after 1024 queries, the growth in non-garbage objects should stop, because it throws away an entry to make room for the new one. That might be correct, the problem here is a leak, that might also affect people reloading Solr in their App-Server. The problem is that somehow on CoreShutdown the MBeans are not destroyed. Maybe this is fine how it is implemented, but when the Solr core shuts down, all statistics data have to be removed and nulled out or whatever. I would revert this patch for now, until there is a better solution. Once you have a new patch, that does not leak those objects on core shutdowns. The horrible thing is that all those huge maps contains stupid strings, all starting with some solr class name (a RequestHandler), followed by an id and various other information. There are (in the dump analyzed with visualvm) millions of String looking like that (OQL executed on heap dump): select count(heap.objects( "java.lang.String" ), 'it.toString().startsWith( "org.apache.solr.handler" )') returns: 305064 (to just show some of the string instances hanging around) This looks like a huge waste!
          Hide
          Mark Miller added a comment -

          I would revert this patch for now

          +1

          Show
          Mark Miller added a comment - I would revert this patch for now +1
          Hide
          Robert Muir added a comment -

          I wonder if the permgen leak is related to the thread leaks caused by this metrics package (see SolrIgnoredThreadsFilter.class, it ignores threads leakd by the metrics package).

          In general we should try to prune this list and keep it minimal... at the moment i'm pretty suspicious of this third party library though: leaking threads, interning strings, etc.

          Show
          Robert Muir added a comment - I wonder if the permgen leak is related to the thread leaks caused by this metrics package (see SolrIgnoredThreadsFilter.class, it ignores threads leakd by the metrics package). In general we should try to prune this list and keep it minimal... at the moment i'm pretty suspicious of this third party library though: leaking threads, interning strings, etc.
          Hide
          Uwe Schindler added a comment -

          Alan Woodward shood revert, if he is not available for the next days, I will do it. OK? (I dont want to be again the bad guy reverting other's commits)

          Show
          Uwe Schindler added a comment - Alan Woodward shood revert, if he is not available for the next days, I will do it. OK? (I dont want to be again the bad guy reverting other's commits)
          Hide
          Robert Muir added a comment -

          A possible patch to the leak (I cannot run solr tests AT ALL on my machine, because it doesnt like my network configuration, sorry, someone else must test).

          The root cause here is Metrics.newXXX. we should not use this, it enrolls entries into a global static MetricsRegistry. So its better to make a throwaway one.

          I think we should still just revert this issue for now. Just explaining where the leak comes from.

          Show
          Robert Muir added a comment - A possible patch to the leak (I cannot run solr tests AT ALL on my machine, because it doesnt like my network configuration, sorry, someone else must test). The root cause here is Metrics.newXXX. we should not use this, it enrolls entries into a global static MetricsRegistry. So its better to make a throwaway one. I think we should still just revert this issue for now. Just explaining where the leak comes from.
          Hide
          Shawn Heisey added a comment -

          Robert, Alan did try creating a registry for each handler. It caused OOM errors when running Solr tests. See comments on 2012/10/26.

          Perhaps we could create a single static registry object that gets used by every handler to create the individual objects. I'll use your patch and attempt that.

          Show
          Shawn Heisey added a comment - Robert, Alan did try creating a registry for each handler. It caused OOM errors when running Solr tests. See comments on 2012/10/26. Perhaps we could create a single static registry object that gets used by every handler to create the individual objects. I'll use your patch and attempt that.
          Hide
          Robert Muir added a comment -

          I dont think we should have a single static: its no better than whats happening right now! its the problem!

          But for now, again I feel the right solution is to back out the change. there are too many resource leaks here.

          Besides the permgen leak, this thing leaks threads like a sieve. and this is currently just being ignored.

          Show
          Robert Muir added a comment - I dont think we should have a single static: its no better than whats happening right now! its the problem! But for now, again I feel the right solution is to back out the change. there are too many resource leaks here. Besides the permgen leak, this thing leaks threads like a sieve. and this is currently just being ignored.
          Hide
          Uwe Schindler added a comment -

          MetricsRegistry is the singleton: With VisualVM you can get the retained size ("Retained size of an object is its shallow size plus the shallow sizes of the objects that are accessible, directly or indirectly, only from this object. In other words, the retained size represents the amount of memory that will be freed by the garbage collector when this object is collected.") -> o holy x-mas! One instance hold references to all metrics ever created. This is the leak, together with the threads spawned in its lifetime.

          Whenever you reload yur Solr instance you leak more objects (which is the classic classloader leak).

          Show
          Uwe Schindler added a comment - MetricsRegistry is the singleton: With VisualVM you can get the retained size ("Retained size of an object is its shallow size plus the shallow sizes of the objects that are accessible, directly or indirectly, only from this object. In other words, the retained size represents the amount of memory that will be freed by the garbage collector when this object is collected.") -> o holy x-mas! One instance hold references to all metrics ever created. This is the leak, together with the threads spawned in its lifetime. Whenever you reload yur Solr instance you leak more objects (which is the classic classloader leak).
          Hide
          Shawn Heisey added a comment -

          I've been trying to find evidence of the leak, and can't. I completely trust that you guys do know what you're talking about, so this is probably just my inexperience.

          What I did find:

          Before any additional patching, looking at jconsole, I have two metrics threads. Looking at MBeans in jconsole, there are 290 metric scopes under RequestHandlerBase with names from metrics-scope-0 to metrics-scope-289.

          Taking Robert's patch and fixing it to create one static MetricsRegistry, the additional objects under MBeans disappear, and the number of threads named "metric-*" goes from 2 to 1. Through some additional debugs, I have files containing stacktraces from every time RequestHandlerBase is called, so I know it is still being called 290 times during startup.

          Examining those stacktraces, I have determined that each of my 16 cores has 18 request handlers in common, with two additional server-wide handlers - CollectionsHandler and CoreAdminHandler - for a total of 290. I'm attaching an archive with all the stacktraces and a summary - the third line from each stacktrace, sorted.

          If we made the metrics package optional on a per-handler basis, I would drop from 290 scopes to 48 - each of my 16 cores has four search handlers, but I really only need full metrics in three of them.

          Show
          Shawn Heisey added a comment - I've been trying to find evidence of the leak, and can't. I completely trust that you guys do know what you're talking about, so this is probably just my inexperience. What I did find: Before any additional patching, looking at jconsole, I have two metrics threads. Looking at MBeans in jconsole, there are 290 metric scopes under RequestHandlerBase with names from metrics-scope-0 to metrics-scope-289. Taking Robert's patch and fixing it to create one static MetricsRegistry, the additional objects under MBeans disappear, and the number of threads named "metric-*" goes from 2 to 1. Through some additional debugs, I have files containing stacktraces from every time RequestHandlerBase is called, so I know it is still being called 290 times during startup. Examining those stacktraces, I have determined that each of my 16 cores has 18 request handlers in common, with two additional server-wide handlers - CollectionsHandler and CoreAdminHandler - for a total of 290. I'm attaching an archive with all the stacktraces and a summary - the third line from each stacktrace, sorted. If we made the metrics package optional on a per-handler basis, I would drop from 290 scopes to 48 - each of my 16 cores has four search handlers, but I really only need full metrics in three of them.
          Hide
          Shawn Heisey added a comment -

          Archive with debug stacktraces and a summary.

          Show
          Shawn Heisey added a comment - Archive with debug stacktraces and a summary.
          Hide
          Shawn Heisey added a comment -

          Some things just clicked and I think I might have figured it out, tell me if I get this wrong. The metrics objects get created during initialization, but then if you reload a core (or possibly if you reload the entire app in your container) then the old metrics objects are never garbage collected because the registry object holds onto them forever.

          I do see that the registry has the ability to remove metrics. To make this possible, RequestHandlerBase would probably need a close() method, and it would need to be properly called when required. I am assuming this is not a trivial undertaking?

          Show
          Shawn Heisey added a comment - Some things just clicked and I think I might have figured it out, tell me if I get this wrong. The metrics objects get created during initialization, but then if you reload a core (or possibly if you reload the entire app in your container) then the old metrics objects are never garbage collected because the registry object holds onto them forever. I do see that the registry has the ability to remove metrics. To make this possible, RequestHandlerBase would probably need a close() method, and it would need to be properly called when required. I am assuming this is not a trivial undertaking?
          Hide
          Shawn Heisey added a comment -

          An extension of Robert's leak.patch, as a possible starting point for a complete solution to the leak problem.

          The patch mimics the overall memory efficiency of the current approach by changing to a static registry object instead of an implied global registry.

          It also implements the Closeable interface, using the now-accessible registry object. This part is incomplete, because calls to the close() method would need to be added to anyplace where a request handler object becomes garbage. I don't know where these places might be.

          Additional note: If we ever extend the metrics capability to things other than request handlers, we will need to make the static registry object global.

          Show
          Shawn Heisey added a comment - An extension of Robert's leak.patch, as a possible starting point for a complete solution to the leak problem. The patch mimics the overall memory efficiency of the current approach by changing to a static registry object instead of an implied global registry. It also implements the Closeable interface, using the now-accessible registry object. This part is incomplete, because calls to the close() method would need to be added to anyplace where a request handler object becomes garbage. I don't know where these places might be. Additional note: If we ever extend the metrics capability to things other than request handlers, we will need to make the static registry object global.
          Hide
          Lance Norskog added a comment -

          The OnlineSummary class in Mahout does the calculations you want. One little class you can steal. No dependencies necessary.

          Show
          Lance Norskog added a comment - The OnlineSummary class in Mahout does the calculations you want. One little class you can steal. No dependencies necessary.
          Hide
          Shawn Heisey added a comment -

          Since I know we have to revert this to keep the 4.1 release clean, I might as well save everyone some time. Here is a patch to revert the latest metrics patch as well as every related mention of yammer and metrics that I could find.

          It's against 4x but applies 100% cleanly to trunk as well.

          All Solr tests pass on 4x.

          In the time that it took for the tests to run, I have realized that changing the fix version to 4.2 may have been premature - probably should have waited until a revert patch was committed.

          Show
          Shawn Heisey added a comment - Since I know we have to revert this to keep the 4.1 release clean, I might as well save everyone some time. Here is a patch to revert the latest metrics patch as well as every related mention of yammer and metrics that I could find. It's against 4x but applies 100% cleanly to trunk as well. All Solr tests pass on 4x. In the time that it took for the tests to run, I have realized that changing the fix version to 4.2 may have been premature - probably should have waited until a revert patch was committed.
          Hide
          Shawn Heisey added a comment -

          Lance, if I am reading OnlineSummarizer right, it only gives you five percentile numbers - 0% (min), 50% (median), 100% (max), and two others that are not explicitly quantified, but I am guessing are 25% and 75%. The 95% and 99% points are not present, and I would argue strongly that those are the most useful numbers currently available. Median is important, but 95% and 99% are the numbers that will show problems first.

          Show
          Shawn Heisey added a comment - Lance, if I am reading OnlineSummarizer right, it only gives you five percentile numbers - 0% (min), 50% (median), 100% (max), and two others that are not explicitly quantified, but I am guessing are 25% and 75%. The 95% and 99% points are not present, and I would argue strongly that those are the most useful numbers currently available. Median is important, but 95% and 99% are the numbers that will show problems first.
          Hide
          Commit Tag Bot added a comment -
          Show
          Commit Tag Bot added a comment - [trunk commit] Uwe Schindler http://svn.apache.org/viewvc?view=revision&revision=1426230 Revert SOLR-1972
          Hide
          Uwe Schindler added a comment -

          Shawn: I reverted the commits related to this isseue (done by romseygeek) in trunk revision: 1426230, merged/reverted 4.x revision: 1426234

          You have now a clean start again!

          Show
          Uwe Schindler added a comment - Shawn: I reverted the commits related to this isseue (done by romseygeek) in trunk revision: 1426230, merged/reverted 4.x revision: 1426234 You have now a clean start again!
          Hide
          Commit Tag Bot added a comment -

          [branch_4x commit] Uwe Schindler
          http://svn.apache.org/viewvc?view=revision&revision=1426234

          Merged revision(s) 1426230 from lucene/dev/trunk:
          Revert SOLR-1972

          Show
          Commit Tag Bot added a comment - [branch_4x commit] Uwe Schindler http://svn.apache.org/viewvc?view=revision&revision=1426234 Merged revision(s) 1426230 from lucene/dev/trunk: Revert SOLR-1972
          Hide
          Lance Norskog added a comment -

          The 25/75 values come from weights, and can be changed to 99/95. I have a patch for that but never submitted it.

          Show
          Lance Norskog added a comment - The 25/75 values come from weights, and can be changed to 99/95. I have a patch for that but never submitted it.
          Hide
          Shawn Heisey added a comment -

          Lance, I see two other problems with OnlineSummarizer.

          1) There are only 100 samples. That's apparently enough for 0/25/50/75/100 quartiles, but it seems very low for 95/99. I was even worried about the 1024 samples in the metrics library, but apparently it actually works very well.

          2) Solr already includes mahout-math 0.3 as a dependency of carrot2. OnlineSummarizer was introduced in 0.4, and there are some serious bugs in it as late as 0.6. Upgrading mahout-math to 0.7 causes clustering (carrot2) tests to fail because it can't find a mahout class. Even the newest version of carrot2 depends specifically on mahout-math 0.3.

          I have a checkout where I am re-applying the metrics patch and have made RequestHandlerBase implement Closeable, but I have zero knowledge about where the close() method would actually have to be called. I have been attempting to figure it out, but Solr is a very large and complex codebase, so I will need some help. I'm almost always idling in #lucene and #solr as 'elyograg' if someone wants to give me some live pointers. I can do the actual work, but I need some help figuring out where handlers are created and destroyed.

          Show
          Shawn Heisey added a comment - Lance, I see two other problems with OnlineSummarizer. 1) There are only 100 samples. That's apparently enough for 0/25/50/75/100 quartiles, but it seems very low for 95/99. I was even worried about the 1024 samples in the metrics library, but apparently it actually works very well. 2) Solr already includes mahout-math 0.3 as a dependency of carrot2. OnlineSummarizer was introduced in 0.4, and there are some serious bugs in it as late as 0.6. Upgrading mahout-math to 0.7 causes clustering (carrot2) tests to fail because it can't find a mahout class. Even the newest version of carrot2 depends specifically on mahout-math 0.3. I have a checkout where I am re-applying the metrics patch and have made RequestHandlerBase implement Closeable, but I have zero knowledge about where the close() method would actually have to be called. I have been attempting to figure it out, but Solr is a very large and complex codebase, so I will need some help. I'm almost always idling in #lucene and #solr as 'elyograg' if someone wants to give me some live pointers. I can do the actual work, but I need some help figuring out where handlers are created and destroyed.
          Hide
          Lance Norskog added a comment -

          2) Solr already includes mahout-math 0.3 as a dependency of carrot2.

          I did not mean to suggest using the Mahout libraries. I would just copy the class source code and change the weights. It has no other dependencies inside the Mahout project.

          Show
          Lance Norskog added a comment - 2) Solr already includes mahout-math 0.3 as a dependency of carrot2. I did not mean to suggest using the Mahout libraries. I would just copy the class source code and change the weights. It has no other dependencies inside the Mahout project.
          Hide
          Alan Woodward added a comment -

          Sorry, have been away from the internet for the past week. Thanks for reverting, Uwe! I'll try and catch up with the thread.

          Show
          Alan Woodward added a comment - Sorry, have been away from the internet for the past week. Thanks for reverting, Uwe! I'll try and catch up with the thread.
          Hide
          Alan Woodward added a comment -

          > Alan did try creating a registry for each handler. It caused OOM errors when running Solr tests. See comments on 2012/10/26.

          Which, in retrospect, ought to have clued me in that something was wrong here...

          I think this is salvageable by having a MetricsRegistry per core, rather than per handler. This then allows the registry to be shut down/nulled out on Core shutdown. I'll try and work up a patch later today.

          FWIW, the latest version of Metrics removes the global registry entirely, presumably to stop idiots like me shooting themselves in the foot in this fashion.

          Show
          Alan Woodward added a comment - > Alan did try creating a registry for each handler. It caused OOM errors when running Solr tests. See comments on 2012/10/26. Which, in retrospect, ought to have clued me in that something was wrong here... I think this is salvageable by having a MetricsRegistry per core, rather than per handler. This then allows the registry to be shut down/nulled out on Core shutdown. I'll try and work up a patch later today. FWIW, the latest version of Metrics removes the global registry entirely, presumably to stop idiots like me shooting themselves in the foot in this fashion.
          Hide
          Shawn Heisey added a comment -

          Alan, my latest plan still utilized the global registry, though it did it explicitly, so that individual metric objects were removed by a close() method on RequestHandlerBase. That would have required many changes - finding every place that a handler might be deleted or replaced. That likely would require significant refactoring rather than just a few code edits. I was prepared to do it, if someone could light the path.

          An explicit registry per core is an awesome idea, if it can be implemented without similar pain to what I've just described. Looking at RequestHandlerBase, I don't see any direct way to access the containing core object. Experimentation shows that adding a constructor with a SolrCore argument is not the right approach because it would require a lot of changes that are also likely to break custom user code.

          Show
          Shawn Heisey added a comment - Alan, my latest plan still utilized the global registry, though it did it explicitly, so that individual metric objects were removed by a close() method on RequestHandlerBase. That would have required many changes - finding every place that a handler might be deleted or replaced. That likely would require significant refactoring rather than just a few code edits. I was prepared to do it, if someone could light the path. An explicit registry per core is an awesome idea, if it can be implemented without similar pain to what I've just described. Looking at RequestHandlerBase, I don't see any direct way to access the containing core object. Experimentation shows that adding a constructor with a SolrCore argument is not the right approach because it would require a lot of changes that are also likely to break custom user code.
          Hide
          Alan Woodward added a comment - - edited

          Cleaning this up is not looking as easy as I had thought, unfortunately. The leaky metrics tick threads have been removed in metrics trunk, but that change isn't in any released version yet. I'll try doing what Lance suggested, and just copy the relevant classes. I think I can then do away with the registry entirely, and just make the Counter and Timer objects final members of RequestHandlerBase. Then we don't get any interned strings or timer threads, but we keep all the nice exponential-decay stats code.

          Metrics is apache 2.0 licensed, so I think I'm OK to just copy classes over?

          Show
          Alan Woodward added a comment - - edited Cleaning this up is not looking as easy as I had thought, unfortunately. The leaky metrics tick threads have been removed in metrics trunk, but that change isn't in any released version yet. I'll try doing what Lance suggested, and just copy the relevant classes. I think I can then do away with the registry entirely, and just make the Counter and Timer objects final members of RequestHandlerBase. Then we don't get any interned strings or timer threads, but we keep all the nice exponential-decay stats code. Metrics is apache 2.0 licensed, so I think I'm OK to just copy classes over?
          Hide
          Uwe Schindler added a comment -

          Metrics is apache 2.0 licensed, so I think I'm OK to just copy classes over?

          Yes, just mention this after the copyright header in the source file that the code was forked.

          Show
          Uwe Schindler added a comment - Metrics is apache 2.0 licensed, so I think I'm OK to just copy classes over? Yes, just mention this after the copyright header in the source file that the code was forked.
          Hide
          Shawn Heisey added a comment - - edited

          elyograg-closeable.patch - my attempt at fixing problems.

          Explicitly uses default registry, includes close() method that removes metrics from the default registry. Sprinkles Closeable around a bit. Turns out it was easier than expected to find places to close() request handlers – SolrCore already includes a close() method, and it conveniently contains a place where all request handlers get registered. I did my best to search in eclipse to make sure I didn't miss something.

          Initially I tried to put "prev.close();" into SolrCore#reload, but that caused test failures. I was able to determine that only CoreContainer#reload currently calls SolrCore#reload, so I now close the old core there, after it registers the new core.

          All tests pass, but I think it would be a good idea to create a test that checks for the thread leak. I don't know if this implementation will solve that problem or not with metrics-core 2.2.0, but I am hopeful.

          I double-checked my implementation with metrics-core 3.0.0-SNAPSHOT. It will require some method name tweaks, but otherwise it is compatible.

          This patch leaves the ThreadLeakFilter that Robert indicated was ridiculous.

          Show
          Shawn Heisey added a comment - - edited elyograg-closeable.patch - my attempt at fixing problems. Explicitly uses default registry, includes close() method that removes metrics from the default registry. Sprinkles Closeable around a bit. Turns out it was easier than expected to find places to close() request handlers – SolrCore already includes a close() method, and it conveniently contains a place where all request handlers get registered. I did my best to search in eclipse to make sure I didn't miss something. Initially I tried to put "prev.close();" into SolrCore#reload, but that caused test failures. I was able to determine that only CoreContainer#reload currently calls SolrCore#reload, so I now close the old core there, after it registers the new core. All tests pass, but I think it would be a good idea to create a test that checks for the thread leak. I don't know if this implementation will solve that problem or not with metrics-core 2.2.0, but I am hopeful. I double-checked my implementation with metrics-core 3.0.0-SNAPSHOT. It will require some method name tweaks, but otherwise it is compatible. This patch leaves the ThreadLeakFilter that Robert indicated was ridiculous.
          Hide
          Alan Woodward added a comment -

          Here's a patch that copies the relevant classes (Counter and Timer and their supporting utility classes) from the metrics library into a new solr package.

          The Counter and Timer objects in RequestHandlerBase are now set at init time (they could probably be final, actually), and there's no concurrenthashmap or registry to use up permgen space. All tests pass. There are a bunch of license headers and javadoc comments to add, but I thought I'd throw this up here first to make sure that people are happy with this approach.

          Show
          Alan Woodward added a comment - Here's a patch that copies the relevant classes (Counter and Timer and their supporting utility classes) from the metrics library into a new solr package. The Counter and Timer objects in RequestHandlerBase are now set at init time (they could probably be final, actually), and there's no concurrenthashmap or registry to use up permgen space. All tests pass. There are a bunch of license headers and javadoc comments to add, but I thought I'd throw this up here first to make sure that people are happy with this approach.
          Hide
          Alan Woodward added a comment -

          Hah, we're duelling patches!

          Show
          Alan Woodward added a comment - Hah, we're duelling patches!
          Hide
          Shawn Heisey added a comment -

          Hah, we're duelling patches!

          May the best approach win, even if it's not mine.

          Show
          Shawn Heisey added a comment - Hah, we're duelling patches! May the best approach win, even if it's not mine.
          Hide
          Uwe Schindler added a comment -

          Hi,
          I would prefer the forked metrics package, because it is cleaner and does not use global static registries or timer threads. It may be possible to remove some classes which are not really needed (e.g. there is a wrapper around an AtomicInteger that adds no functionality, just wraps all methods) - we could maybe remove this and use the original class from JDK. This all is just simple statistics and we need only some classes keeping track of requests, which are owned by the RequestHandler.

          I am not so happy with the metrics dependency at all, because it does too much and uncontrollable.

          Show
          Uwe Schindler added a comment - Hi, I would prefer the forked metrics package, because it is cleaner and does not use global static registries or timer threads. It may be possible to remove some classes which are not really needed (e.g. there is a wrapper around an AtomicInteger that adds no functionality, just wraps all methods) - we could maybe remove this and use the original class from JDK. This all is just simple statistics and we need only some classes keeping track of requests, which are owned by the RequestHandler. I am not so happy with the metrics dependency at all, because it does too much and uncontrollable.
          Hide
          Alan Woodward added a comment -

          Thanks Uwe. I'll clean up the patch and add the javadocs and licenses.

          Shawn, will you have a chance to try my latest patch? You're acting as a very useful guinea pig for this issue.

          Show
          Alan Woodward added a comment - Thanks Uwe. I'll clean up the patch and add the javadocs and licenses. Shawn, will you have a chance to try my latest patch? You're acting as a very useful guinea pig for this issue.
          Hide
          Shawn Heisey added a comment - - edited

          Building now. When tests pass and the build finishes, I will give it a test spin.

          Although I think it would be nice to use metrics directly, I also completely understand and agree with Uwe's points, so I won't object. This still results in the features I need, and it may even happen in time for 4.1.

          Show
          Shawn Heisey added a comment - - edited Building now. When tests pass and the build finishes, I will give it a test spin. Although I think it would be nice to use metrics directly, I also completely understand and agree with Uwe's points, so I won't object. This still results in the features I need, and it may even happen in time for 4.1.
          Hide
          Shawn Heisey added a comment -

          Alan,

          Operationally, looks like everything is good with your new patch on branch_4x. I have a homegrown servlet that utilizes the new stats, and it seems to work perfectly. I am pounding on one of my search handlers and the stats are updating nicely.

          All lucene and solr tests passed. I presume that the leaks that cropped up before will not be present - but I'll let you guys figure that out!

          Show
          Shawn Heisey added a comment - Alan, Operationally, looks like everything is good with your new patch on branch_4x. I have a homegrown servlet that utilizes the new stats, and it seems to work perfectly. I am pounding on one of my search handlers and the stats are updating nicely. All lucene and solr tests passed. I presume that the leaks that cropped up before will not be present - but I'll let you guys figure that out!
          Hide
          Alan Woodward added a comment -

          Updated patch, removing ThreadLocalRandom in favour of just using java.util.Random; plus added license headers and reformatted to solr standard.

          I think this is ready to go in

          Show
          Alan Woodward added a comment - Updated patch, removing ThreadLocalRandom in favour of just using java.util.Random; plus added license headers and reformatted to solr standard. I think this is ready to go in
          Hide
          David Smiley added a comment -

          Alan, I recommend that you report the problems discovered in the Metrics package upstream (i.e. file a github issue), and while you're at it give a link back to this JIRA issue. Hopefully they will be fixed upstream, which is a good thing for open-source in general even if we choose to keep a fork.

          Show
          David Smiley added a comment - Alan, I recommend that you report the problems discovered in the Metrics package upstream (i.e. file a github issue), and while you're at it give a link back to this JIRA issue. Hopefully they will be fixed upstream, which is a good thing for open-source in general even if we choose to keep a fork.
          Hide
          Alan Woodward added a comment -

          Hi David, this has actually already been fixed upstream, or at least the leaky threads bit has been. It just hasn't been released yet. I'll report the MetricsRegistry lifecycle issues though.

          Show
          Alan Woodward added a comment - Hi David, this has actually already been fixed upstream, or at least the leaky threads bit has been. It just hasn't been released yet. I'll report the MetricsRegistry lifecycle issues though.
          Hide
          Shawn Heisey added a comment -

          Alan, updated to your latest patch. Everything is still working very well.

          Show
          Shawn Heisey added a comment - Alan, updated to your latest patch. Everything is still working very well.
          Hide
          Alan Woodward added a comment -

          I've committed the latest patch. Will leave it a couple of days before resolving to make sure this doesn't blow up in an excitingly new and different way.

          Show
          Alan Woodward added a comment - I've committed the latest patch. Will leave it a couple of days before resolving to make sure this doesn't blow up in an excitingly new and different way.
          Hide
          Commit Tag Bot added a comment -

          [branch_4x commit] Alan Woodward
          http://svn.apache.org/viewvc?view=revision&revision=1428381

          SOLR-1972: Add extra query stats to RequestHandler

          Show
          Commit Tag Bot added a comment - [branch_4x commit] Alan Woodward http://svn.apache.org/viewvc?view=revision&revision=1428381 SOLR-1972 : Add extra query stats to RequestHandler
          Hide
          Commit Tag Bot added a comment -

          [trunk commit] Alan Woodward
          http://svn.apache.org/viewvc?view=revision&revision=1428372

          SOLR-1972: Add extra query stats to RequestHandler

          Show
          Commit Tag Bot added a comment - [trunk commit] Alan Woodward http://svn.apache.org/viewvc?view=revision&revision=1428372 SOLR-1972 : Add extra query stats to RequestHandler
          Hide
          Steve Rowe added a comment -

          Alan, can this be resolved as fixed now?

          Show
          Steve Rowe added a comment - Alan, can this be resolved as fixed now?
          Hide
          Commit Tag Bot added a comment -

          [branch_4x commit] Alan Woodward
          http://svn.apache.org/viewvc?view=revision&revision=1413797

          SOLR-1972: Add extra query stats to RequestHandler

          Show
          Commit Tag Bot added a comment - [branch_4x commit] Alan Woodward http://svn.apache.org/viewvc?view=revision&revision=1413797 SOLR-1972 : Add extra query stats to RequestHandler

            People

            • Assignee:
              Alan Woodward
              Reporter:
              Shawn Heisey
            • Votes:
              6 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development