Solr
  1. Solr
  2. SOLR-2554

RandomSortField values are cached in the FieldCache

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.1
    • Fix Version/s: 3.3
    • Component/s: search
    • Labels:
      None

      Description

      The values of RandomSortField get cached in the FieldCache. When using many RandomSortFields over time, this leads to running out of memory.

      This may be one of the cases already covered in SOLR-1111 but I'm not sure.

        Activity

        Hide
        Hoss Man added a comment -

        Hmm.... reviewing the code i don't see any way RandomSortField would use the FieldCache. (or ever could have in any previous release)

        I did some very basic testing with the example solr configs on trunk and i can not reproduce...

        starting solr up clean, loading the sample data and then executing these queries...

        ...i got three different orderings, but when i then checked http://localhost:8983/solr/admin/stats.jsp#cache i verified that fieldCache was empty.

        If you get different results, please re-open and be specific about the version of solr you are using, the steps to reproduce, and the info about fieldCache that you get back from stats.jsp

        Show
        Hoss Man added a comment - Hmm.... reviewing the code i don't see any way RandomSortField would use the FieldCache. (or ever could have in any previous release) I did some very basic testing with the example solr configs on trunk and i can not reproduce... starting solr up clean, loading the sample data and then executing these queries... http://localhost:8983/solr/select/?q=*%3A*&sort=random_foo+asc http://localhost:8983/solr/select/?q=*%3A*&sort=random_bar+asc http://localhost:8983/solr/select/?q=*%3A*&sort=random_yak+asc ...i got three different orderings, but when i then checked http://localhost:8983/solr/admin/stats.jsp#cache i verified that fieldCache was empty. If you get different results, please re-open and be specific about the version of solr you are using, the steps to reproduce, and the info about fieldCache that you get back from stats.jsp
        Hide
        Yonik Seeley added a comment -

        The reporter probably meant the filterCache (although the filterCache should be sized to avoid OOM errors).
        Anyway, I plan on starting work soon on a "cache=false" option for queries.

        Show
        Yonik Seeley added a comment - The reporter probably meant the filterCache (although the filterCache should be sized to avoid OOM errors). Anyway, I plan on starting work soon on a "cache=false" option for queries.
        Hide
        Vadim Geshel added a comment -

        Sorry, I should have been more specific. This happens if you use a RandomSortField in a query, not as a sort criterion:

        http://localhost:8983/solr/select/?q=

        {!func}

        random_foo

        You should immediately see this in stats.jsp#cache, I see this:

        entry#1 : 'org.apache.lucene.store.MMapDirectory$MMapIndexInput@37f02eaa'=>'random_foo',class org.apache.lucene.search.FieldCache$StringIndex,null=>org.apache.lucene.search.FieldCache$StringIndex#2138852435

        I'm using Solr 3.1

        Show
        Vadim Geshel added a comment - Sorry, I should have been more specific. This happens if you use a RandomSortField in a query, not as a sort criterion: http://localhost:8983/solr/select/?q= {!func} random_foo You should immediately see this in stats.jsp#cache, I see this: entry#1 : 'org.apache.lucene.store.MMapDirectory$MMapIndexInput@37f02eaa'=>'random_foo',class org.apache.lucene.search.FieldCache$StringIndex,null=>org.apache.lucene.search.FieldCache$StringIndex#2138852435 I'm using Solr 3.1
        Hide
        Yonik Seeley added a comment -

        Thanks Vadim, I can reproduce this on 3.1 and branch_3x (but trunk seems to work fine), and I'll look into fixing it tomorrow.

        Show
        Yonik Seeley added a comment - Thanks Vadim, I can reproduce this on 3.1 and branch_3x (but trunk seems to work fine), and I'll look into fixing it tomorrow.
        Hide
        Robert Muir added a comment -

        Bulk close for 3.3

        Show
        Robert Muir added a comment - Bulk close for 3.3

          People

          • Assignee:
            Unassigned
            Reporter:
            Vadim Geshel
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development