Lucene - Core
  1. Lucene - Core
  2. LUCENE-2093

Use query-private scope instead of shared Term->TermInfo cache

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.0, 5.0
    • Component/s: core/search
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Spinoff of LUCENE-2075.

      We currently use a shared terms cache so multiple resolves of the same term within execution of a single query save CPU. But this ties up a good amount of long term RAM...

      So, it might be better to instead create a "query private scope", where places in Lucene like the terms dict could store & retrieve results. The scope would be private to each running query, and would be GCable as soon as the query completes. Then we've have perfect within query hit rate...

        Activity

        Hide
        Michael McCandless added a comment -

        If we don't do this in 3.1, we should at least drop the size of the terms dict cache – by rough math, that cache will consume 4 MB on a 20 segment index, even for a smallish index.

        When flex lands, the cache is no longer beneficial for automaton query so it need not be so large.

        Show
        Michael McCandless added a comment - If we don't do this in 3.1, we should at least drop the size of the terms dict cache – by rough math, that cache will consume 4 MB on a 20 segment index, even for a smallish index. When flex lands, the cache is no longer beneficial for automaton query so it need not be so large.
        Hide
        Michael McCandless added a comment -

        We've improved queries so they now save their own term state during rewrite and re-use it during matching.

        Show
        Michael McCandless added a comment - We've improved queries so they now save their own term state during rewrite and re-use it during matching.
        Hide
        Uwe Schindler added a comment -

        Closed after release.

        Show
        Uwe Schindler added a comment - Closed after release.

          People

          • Assignee:
            Unassigned
            Reporter:
            Michael McCandless
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development