Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-1195

Performance improvement for TermInfosReader

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.4
    • Component/s: core/index
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Currently we have a bottleneck for multi-term queries: the dictionary lookup is being done
      twice for each term. The first time in Similarity.idf(), where searcher.docFreq() is called.
      The second time when the posting list is opened (TermDocs or TermPositions).

      The dictionary lookup is not cheap, that's why a significant performance improvement is
      possible here if we avoid the second lookup. An easy way to do this is to add a small LRU
      cache to TermInfosReader.

      I ran some performance experiments with an LRU cache size of 20, and an mid-size index of
      500,000 documents from wikipedia. Here are some test results:

      50,000 AND queries with 3 terms each:
      old: 152 secs
      new (with LRU cache): 112 secs (26% faster)

      50,000 OR queries with 3 terms each:
      old: 175 secs
      new (with LRU cache): 133 secs (24% faster)

      For bigger indexes this patch will probably have less impact, for smaller once more.

      I will attach a patch soon.

        Attachments

        1. lucene-1195.patch
          4 kB
          Michael Busch
        2. lucene-1195.patch
          15 kB
          Michael Busch
        3. lucene-1195.patch
          19 kB
          Michael Busch
        4. SafeThreadLocal.java
          1 kB
          robert engels

          Activity

            People

            • Assignee:
              michaelbusch Michael Busch
              Reporter:
              michaelbusch Michael Busch
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: