Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-2075

Share the Term -> TermInfo cache across threads

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments


    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.0-ALPHA
    • Component/s: core/index
    • Labels:
    • Lucene Fields:


      Right now each thread creates its own (thread private) SimpleLRUCache,
      holding up to 1024 terms.

      This is rather wasteful, since if there are a high number of threads
      that come through Lucene, you're multiplying the RAM usage. You're
      also cutting way back on likelihood of a cache hit (except the known
      multiple times we lookup a term within-query, which uses one thread).
      In NRT search we open new SegmentReaders (on tiny segments) often
      which each thread must then spend CPU/RAM creating & populating.

      Now that we are on 1.5 we can use java.util.concurrent.*, eg
      ConcurrentHashMap. One simple approach could be a double-barrel LRU
      cache, using 2 maps (primary, secondary). You check the cache by
      first checking primary; if that's a miss, you check secondary and if
      you get a hit you promote it to primary. Once primary is full you
      clear secondary and swap them.

      Or... any other suggested approach?


        1. ConcurrentLRUCache.java
          20 kB
          Mark Miller
        2. LUCENE-2075.patch
          21 kB
          Michael McCandless
        3. LUCENE-2075.patch
          20 kB
          Michael McCandless
        4. LUCENE-2075.patch
          19 kB
          Michael McCandless
        5. LUCENE-2075.patch
          12 kB
          Michael McCandless
        6. LUCENE-2075.patch
          12 kB
          Uwe Schindler
        7. LUCENE-2075.patch
          12 kB
          Michael McCandless
        8. LUCENE-2075.patch
          25 kB
          Michael McCandless
        9. LUCENE-2075.patch
          16 kB
          Uwe Schindler
        10. LUCENE-2075.patch
          16 kB
          Uwe Schindler
        11. LUCENE-2075.patch
          15 kB
          Yonik Seeley
        12. LUCENE-2075.patch
          15 kB
          Yonik Seeley
        13. LUCENE-2075.patch
          15 kB
          Yonik Seeley



            • Assignee:
              mikemccand Michael McCandless
              mikemccand Michael McCandless


              • Created:

                Issue deployment