Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-2093

Use query-private scope instead of shared Term->TermInfo cache

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.0, 6.0
    • Component/s: core/search
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Spinoff of LUCENE-2075.

      We currently use a shared terms cache so multiple resolves of the same term within execution of a single query save CPU. But this ties up a good amount of long term RAM...

      So, it might be better to instead create a "query private scope", where places in Lucene like the terms dict could store & retrieve results. The scope would be private to each running query, and would be GCable as soon as the query completes. Then we've have perfect within query hit rate...

        Activity

        Hide
        mikemccand Michael McCandless added a comment -

        If we don't do this in 3.1, we should at least drop the size of the terms dict cache – by rough math, that cache will consume 4 MB on a 20 segment index, even for a smallish index.

        When flex lands, the cache is no longer beneficial for automaton query so it need not be so large.

        Show
        mikemccand Michael McCandless added a comment - If we don't do this in 3.1, we should at least drop the size of the terms dict cache – by rough math, that cache will consume 4 MB on a 20 segment index, even for a smallish index. When flex lands, the cache is no longer beneficial for automaton query so it need not be so large.
        Hide
        mikemccand Michael McCandless added a comment -

        We've improved queries so they now save their own term state during rewrite and re-use it during matching.

        Show
        mikemccand Michael McCandless added a comment - We've improved queries so they now save their own term state during rewrite and re-use it during matching.
        Hide
        thetaphi Uwe Schindler added a comment -

        Closed after release.

        Show
        thetaphi Uwe Schindler added a comment - Closed after release.

          People

          • Assignee:
            Unassigned
            Reporter:
            mikemccand Michael McCandless
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development