Solr
  1. Solr
  2. SOLR-1292

show lucene fieldcache entries and sizes

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.4
    • Component/s: None
    • Labels:
      None

      Description

      See LUCENE-1749, FieldCache introspection API

        Issue Links

          Activity

          Hide
          Grant Ingersoll added a comment -

          Given our goal is to turn around Solr 1.4 fairly quickly after 2.9 and the fact that this issue doesn't have a patch, should we mark it as 1.5?

          Show
          Grant Ingersoll added a comment - Given our goal is to turn around Solr 1.4 fairly quickly after 2.9 and the fact that this issue doesn't have a patch, should we mark it as 1.5?
          Hide
          Yonik Seeley added a comment -

          Depends on how quickly we can get it in.
          I hate to see good functionality put in Lucene that can't be used in Solr, and this should help us (and others) debug any fieldcache issues.

          Show
          Yonik Seeley added a comment - Depends on how quickly we can get it in. I hate to see good functionality put in Lucene that can't be used in Solr, and this should help us (and others) debug any fieldcache issues.
          Hide
          Mark Miller added a comment -

          Yeah - I'd wait to push this - its an easy one to push last second if we have to. 2.9 is not out yet, and if a good patch is up soon - why not put it in?

          Its important here for similar reasons that its important to get in Lucene 2.9.

          I'd been waiting for the issue to get committed to start (it relies on functionality in Lucene that is not there in Solr yet)

          But really there is no need to wait - I had just started thinking about the best way to go about it.

          Show
          Mark Miller added a comment - Yeah - I'd wait to push this - its an easy one to push last second if we have to. 2.9 is not out yet, and if a good patch is up soon - why not put it in? Its important here for similar reasons that its important to get in Lucene 2.9. I'd been waiting for the issue to get committed to start (it relies on functionality in Lucene that is not there in Solr yet) But really there is no need to wait - I had just started thinking about the best way to go about it.
          Hide
          Grant Ingersoll added a comment -

          OK, definitely agree, was just going through the 1.4 unresolved list and this one had no progress, so I thought I would ask.

          Show
          Grant Ingersoll added a comment - OK, definitely agree, was just going through the 1.4 unresolved list and this one had no progress, so I thought I would ask.
          Hide
          Hoss Man added a comment -

          rough start ... adds a hardcoded MBean that just wraps FieldCache.getEntries() and FieldCacheSanityChecker in it's getStatistics() method.

          Show
          Hoss Man added a comment - rough start ... adds a hardcoded MBean that just wraps FieldCache.getEntries() and FieldCacheSanityChecker in it's getStatistics() method.
          Hide
          Hoss Man added a comment -

          this does everything the issue summary suggests in a very simple way ... but personally i think we should also use FieldCache.setInfoStream and and log an error anytime anything is written to that stream – but i'm not sure if it's worth adding an option for that ... if it was always on it could (concievably) have a noticeable effect on static warming of many small field caches.

          Show
          Hoss Man added a comment - this does everything the issue summary suggests in a very simple way ... but personally i think we should also use FieldCache.setInfoStream and and log an error anytime anything is written to that stream – but i'm not sure if it's worth adding an option for that ... if it was always on it could (concievably) have a noticeable effect on static warming of many small field caches.
          Hide
          Jason Rutherglen added a comment -

          Does this include the memory size of UnInvertedField?

          Show
          Jason Rutherglen added a comment - Does this include the memory size of UnInvertedField?
          Hide
          Yonik Seeley added a comment -

          Does this include the memory size of UnInvertedField?

          UnInvertedField already did it's own size calculation (along with other stats about number of terms).

          Show
          Yonik Seeley added a comment - Does this include the memory size of UnInvertedField? UnInvertedField already did it's own size calculation (along with other stats about number of terms).
          Hide
          Jason Rutherglen added a comment -

          I did an open call hierarchy on UnInvertedField.memSize, which is called by toString which isn't called by anything so I assume it never makes it to the web UI?

          Show
          Jason Rutherglen added a comment - I did an open call hierarchy on UnInvertedField.memSize, which is called by toString which isn't called by anything so I assume it never makes it to the web UI?
          Hide
          Grant Ingersoll added a comment -

          Looks like a reasonable first pass, Hoss. I think we should commit and then we can iterate in 1.5.

          Show
          Grant Ingersoll added a comment - Looks like a reasonable first pass, Hoss. I think we should commit and then we can iterate in 1.5.
          Hide
          Hoss Man added a comment -

          This looks like as far as we'll get on this for 1.4...

          Committed revision 818618.

          Show
          Hoss Man added a comment - This looks like as far as we'll get on this for 1.4... Committed revision 818618.
          Hide
          Shalin Shekhar Mangar added a comment -

          I did an open call hierarchy on UnInvertedField.memSize, which is called by toString which isn't called by anything so I assume it never makes it to the web UI?

          Jason is right. The UninvertedField's memSize never shows up on the statistics page. I'll open another issue to fix this.

          Show
          Shalin Shekhar Mangar added a comment - I did an open call hierarchy on UnInvertedField.memSize, which is called by toString which isn't called by anything so I assume it never makes it to the web UI? Jason is right. The UninvertedField's memSize never shows up on the statistics page. I'll open another issue to fix this.
          Hide
          Shalin Shekhar Mangar added a comment -

          I opened SOLR-1461

          Show
          Shalin Shekhar Mangar added a comment - I opened SOLR-1461
          Hide
          Hoss Man added a comment -

          Followup note...

          hossman@brunner:~/lucene/solr$ svn commit -m "SOLR-1292: followup tweak based on mailing list discussion: for indexes with lots of segments, estimating the size of all the field caches can take a lot of time (particularly with lots of String based caches) so for now we only estimate the size for items which are part of instanity instances"
          Sending src/java/org/apache/solr/search/SolrFieldCacheMBean.java
          Transmitting file data .
          Committed revision 826788.

          Show
          Hoss Man added a comment - Followup note... hossman@brunner:~/lucene/solr$ svn commit -m " SOLR-1292 : followup tweak based on mailing list discussion: for indexes with lots of segments, estimating the size of all the field caches can take a lot of time (particularly with lots of String based caches) so for now we only estimate the size for items which are part of instanity instances" Sending src/java/org/apache/solr/search/SolrFieldCacheMBean.java Transmitting file data . Committed revision 826788.
          Hide
          Grant Ingersoll added a comment -

          Bulk close for Solr 1.4

          Show
          Grant Ingersoll added a comment - Bulk close for Solr 1.4

            People

            • Assignee:
              Hoss Man
              Reporter:
              Yonik Seeley
            • Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development