Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-1536

if a filter can support random access API, we should use it

    Details

    • Lucene Fields:
      New

      Description

      I ran some performance tests, comparing applying a filter via
      random-access API instead of current trunk's iterator API.

      This was inspired by LUCENE-1476, where we realized deletions should
      really be implemented just like a filter, but then in testing found
      that switching deletions to iterator was a very sizable performance
      hit.

      Some notes on the test:

      • Index is first 2M docs of Wikipedia. Test machine is Mac OS X
        10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
      • I test across multiple queries. 1-X means an OR query, eg 1-4
        means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
        AND 3 AND 4. "u s" means "united states" (phrase search).
      • I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
        95, 98, 99, 99.99999 (filter is non-null but all bits are set),
        100 (filter=null, control)).
      • Method high means I use random-access filter API in
        IndexSearcher's main loop. Method low means I use random-access
        filter API down in SegmentTermDocs (just like deleted docs
        today).
      • Baseline (QPS) is current trunk, where filter is applied as iterator up
        "high" (ie in IndexSearcher's search loop).

        Attachments

        1. CachedFilterIndexReader.java
          4 kB
          Michael McCandless
        2. changes-yonik-uwe.patch
          3 kB
          Uwe Schindler
        3. LUCENE-1536_hack.patch
          150 kB
          Robert Muir
        4. LUCENE-1536.patch
          52 kB
          Uwe Schindler
        5. LUCENE-1536.patch
          148 kB
          Uwe Schindler
        6. LUCENE-1536.patch
          148 kB
          Uwe Schindler
        7. LUCENE-1536.patch
          148 kB
          Uwe Schindler
        8. LUCENE-1536.patch
          147 kB
          Uwe Schindler
        9. LUCENE-1536.patch
          147 kB
          Robert Muir
        10. LUCENE-1536.patch
          141 kB
          Uwe Schindler
        11. LUCENE-1536.patch
          141 kB
          Yonik Seeley
        12. LUCENE-1536.patch
          123 kB
          Robert Muir
        13. LUCENE-1536.patch
          52 kB
          Robert Muir
        14. LUCENE-1536.patch
          29 kB
          Uwe Schindler
        15. LUCENE-1536.patch
          28 kB
          Uwe Schindler
        16. LUCENE-1536.patch
          28 kB
          Uwe Schindler
        17. LUCENE-1536.patch
          24 kB
          Chris Male
        18. LUCENE-1536.patch
          24 kB
          Robert Muir
        19. LUCENE-1536.patch
          20 kB
          Robert Muir
        20. LUCENE-1536.patch
          24 kB
          Chris Male
        21. LUCENE-1536.patch
          75 kB
          Michael McCandless
        22. LUCENE-1536.patch
          71 kB
          Michael McCandless
        23. LUCENE-1536.patch
          68 kB
          Chris Male
        24. LUCENE-1536.patch
          67 kB
          Chris Male
        25. LUCENE-1536.patch
          69 kB
          Chris Male
        26. LUCENE-1536.patch
          63 kB
          Michael McCandless
        27. LUCENE-1536.patch
          199 kB
          Michael McCandless
        28. LUCENE-1536.patch
          55 kB
          Jason Rutherglen
        29. LUCENE-1536.patch
          51 kB
          Jason Rutherglen
        30. LUCENE-1536.patch
          26 kB
          Jason Rutherglen
        31. LUCENE-1536.patch
          19 kB
          Jason Rutherglen
        32. LUCENE-1536.patch
          11 kB
          Michael McCandless
        33. LUCENE-1536-rewrite.patch
          0.1 kB
          Uwe Schindler
        34. LUCENE-1536-rewrite.patch
          137 kB
          Uwe Schindler
        35. LUCENE-1536-rewrite.patch
          128 kB
          Uwe Schindler
        36. LUCENE-1536-rewrite.patch
          115 kB
          Uwe Schindler
        37. LUCENE-1536-rewrite.patch
          88 kB
          Uwe Schindler
        38. LUCENE-1536-rewrite.patch
          84 kB
          Uwe Schindler
        39. LUCENE-1536-rewrite.patch
          71 kB
          Uwe Schindler
        40. LUCENE-1536-rewrite.patch
          52 kB
          Uwe Schindler
        41. luceneutil.patch
          3 kB
          Robert Muir

          Issue Links

            Activity

              People

              • Assignee:
                thetaphi Uwe Schindler
                Reporter:
                mikemccand Michael McCandless
              • Votes:
                2 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: