Lucene - Core
  1. Lucene - Core
  2. LUCENE-1536

if a filter can support random access API, we should use it

    Details

    • Lucene Fields:
      New

      Description

      I ran some performance tests, comparing applying a filter via
      random-access API instead of current trunk's iterator API.

      This was inspired by LUCENE-1476, where we realized deletions should
      really be implemented just like a filter, but then in testing found
      that switching deletions to iterator was a very sizable performance
      hit.

      Some notes on the test:

      • Index is first 2M docs of Wikipedia. Test machine is Mac OS X
        10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
      • I test across multiple queries. 1-X means an OR query, eg 1-4
        means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
        AND 3 AND 4. "u s" means "united states" (phrase search).
      • I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
        95, 98, 99, 99.99999 (filter is non-null but all bits are set),
        100 (filter=null, control)).
      • Method high means I use random-access filter API in
        IndexSearcher's main loop. Method low means I use random-access
        filter API down in SegmentTermDocs (just like deleted docs
        today).
      • Baseline (QPS) is current trunk, where filter is applied as iterator up
        "high" (ie in IndexSearcher's search loop).
      1. LUCENE-1536.patch
        11 kB
        Michael McCandless
      2. LUCENE-1536.patch
        19 kB
        Jason Rutherglen
      3. LUCENE-1536.patch
        26 kB
        Jason Rutherglen
      4. LUCENE-1536.patch
        51 kB
        Jason Rutherglen
      5. LUCENE-1536.patch
        55 kB
        Jason Rutherglen
      6. CachedFilterIndexReader.java
        4 kB
        Michael McCandless
      7. LUCENE-1536.patch
        199 kB
        Michael McCandless
      8. LUCENE-1536.patch
        63 kB
        Michael McCandless
      9. LUCENE-1536.patch
        69 kB
        Chris Male
      10. LUCENE-1536.patch
        67 kB
        Chris Male
      11. LUCENE-1536.patch
        68 kB
        Chris Male
      12. LUCENE-1536.patch
        71 kB
        Michael McCandless
      13. LUCENE-1536.patch
        75 kB
        Michael McCandless
      14. LUCENE-1536.patch
        24 kB
        Chris Male
      15. LUCENE-1536.patch
        20 kB
        Robert Muir
      16. LUCENE-1536.patch
        24 kB
        Robert Muir
      17. LUCENE-1536.patch
        24 kB
        Chris Male
      18. LUCENE-1536.patch
        28 kB
        Uwe Schindler
      19. LUCENE-1536.patch
        28 kB
        Uwe Schindler
      20. LUCENE-1536.patch
        29 kB
        Uwe Schindler
      21. LUCENE-1536-rewrite.patch
        52 kB
        Uwe Schindler
      22. LUCENE-1536.patch
        52 kB
        Robert Muir
      23. LUCENE-1536-rewrite.patch
        71 kB
        Uwe Schindler
      24. LUCENE-1536-rewrite.patch
        84 kB
        Uwe Schindler
      25. LUCENE-1536-rewrite.patch
        88 kB
        Uwe Schindler
      26. LUCENE-1536-rewrite.patch
        115 kB
        Uwe Schindler
      27. LUCENE-1536.patch
        123 kB
        Robert Muir
      28. LUCENE-1536-rewrite.patch
        128 kB
        Uwe Schindler
      29. LUCENE-1536-rewrite.patch
        137 kB
        Uwe Schindler
      30. luceneutil.patch
        3 kB
        Robert Muir
      31. LUCENE-1536.patch
        141 kB
        Yonik Seeley
      32. LUCENE-1536-rewrite.patch
        0.1 kB
        Uwe Schindler
      33. LUCENE-1536.patch
        141 kB
        Uwe Schindler
      34. changes-yonik-uwe.patch
        3 kB
        Uwe Schindler
      35. LUCENE-1536.patch
        147 kB
        Robert Muir
      36. LUCENE-1536.patch
        147 kB
        Uwe Schindler
      37. LUCENE-1536.patch
        148 kB
        Uwe Schindler
      38. LUCENE-1536_hack.patch
        150 kB
        Robert Muir
      39. LUCENE-1536.patch
        148 kB
        Uwe Schindler
      40. LUCENE-1536.patch
        148 kB
        Uwe Schindler
      41. LUCENE-1536.patch
        52 kB
        Uwe Schindler

        Issue Links

          Activity

          Uwe Schindler made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Uwe Schindler made changes -
          Link This issue is related to LUCENE-4548 [ LUCENE-4548 ]
          Yonik Seeley made changes -
          Link This issue breaks SOLR-3062 [ SOLR-3062 ]
          Uwe Schindler made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Uwe Schindler made changes -
          Assignee Michael McCandless [ mikemccand ] Uwe Schindler [ thetaphi ]
          Uwe Schindler made changes -
          Attachment LUCENE-1536.patch [ 12500433 ]
          Uwe Schindler made changes -
          Attachment LUCENE-1536.patch [ 12498865 ]
          Uwe Schindler made changes -
          Attachment LUCENE-1536.patch [ 12498579 ]
          Uwe Schindler made changes -
          Attachment LUCENE-1536.patch [ 12498578 ]
          Uwe Schindler made changes -
          Attachment LUCENE-1536.patch [ 12498578 ]
          Uwe Schindler made changes -
          Attachment LUCENE-1536.patch [ 12498530 ]
          Uwe Schindler made changes -
          Attachment LUCENE-1536.patch [ 12498530 ]
          Robert Muir made changes -
          Link This issue is blocked by LUCENE-3503 [ LUCENE-3503 ]
          Robert Muir made changes -
          Attachment LUCENE-1536_hack.patch [ 12498440 ]
          Uwe Schindler made changes -
          Attachment LUCENE-1536.patch [ 12498326 ]
          Uwe Schindler made changes -
          Attachment LUCENE-1536.patch [ 12498317 ]
          Robert Muir made changes -
          Attachment LUCENE-1536.patch [ 12498314 ]
          Uwe Schindler made changes -
          Attachment LUCENE-1536-rewrite.patch [ 12498287 ]
          Attachment LUCENE-1536.patch [ 12498288 ]
          Attachment changes-yonik-uwe.patch [ 12498289 ]
          Yonik Seeley made changes -
          Attachment LUCENE-1536.patch [ 12498253 ]
          Robert Muir made changes -
          Attachment luceneutil.patch [ 12498229 ]
          Uwe Schindler made changes -
          Attachment LUCENE-1536-rewrite.patch [ 12498173 ]
          Uwe Schindler made changes -
          Attachment LUCENE-1536-rewrite.patch [ 12498158 ]
          Robert Muir made changes -
          Attachment LUCENE-1536.patch [ 12498150 ]
          Uwe Schindler made changes -
          Attachment LUCENE-1536-rewrite.patch [ 12498135 ]
          Uwe Schindler made changes -
          Attachment LUCENE-1536-rewrite.patch [ 12498090 ]
          Uwe Schindler made changes -
          Attachment LUCENE-1536-rewrite.patch [ 12498079 ]
          Uwe Schindler made changes -
          Attachment LUCENE-1536-rewrite.patch [ 12497996 ]
          Robert Muir made changes -
          Attachment LUCENE-1536.patch [ 12497985 ]
          Uwe Schindler made changes -
          Attachment LUCENE-1536-rewrite.patch [ 12497980 ]
          Uwe Schindler made changes -
          Attachment LUCENE-1536.patch [ 12497887 ]
          Uwe Schindler made changes -
          Attachment LUCENE-1536.patch [ 12497867 ]
          Uwe Schindler made changes -
          Attachment LUCENE-1536.patch [ 12497863 ]
          Chris Male made changes -
          Attachment LUCENE-1536.patch [ 12497740 ]
          Robert Muir made changes -
          Attachment LUCENE-1536.patch [ 12497689 ]
          Robert Muir made changes -
          Attachment LUCENE-1536.patch [ 12497680 ]
          Chris Male made changes -
          Attachment LUCENE-1536.patch [ 12497400 ]
          Michael McCandless made changes -
          Attachment LUCENE-1536.patch [ 12496874 ]
          Michael McCandless made changes -
          Attachment LUCENE-1536.patch [ 12496781 ]
          Chris Male made changes -
          Attachment LUCENE-1536.patch [ 12496664 ]
          Chris Male made changes -
          Attachment LUCENE-1536.patch [ 12496227 ]
          Chris Male made changes -
          Attachment LUCENE-1536.patch [ 12496119 ]
          Michael McCandless made changes -
          Attachment LUCENE-1536.patch [ 12485822 ]
          Uwe Schindler made changes -
          Link This issue relates to LUCENE-3212 [ LUCENE-3212 ]
          Michael McCandless made changes -
          Attachment LUCENE-1536.patch [ 12483869 ]
          Simon Willnauer made changes -
          Labels gsoc2011, lucene-gsoc-11 mentor, gsoc2011 lucene-gsoc-11 mentor
          Simon Willnauer made changes -
          Labels mentor gsoc2011, lucene-gsoc-11 mentor,
          Simon Willnauer made changes -
          Labels gsoc2011 lucene-gsoc-11 mentor
          Michael McCandless made changes -
          Labels gsoc2011 lucene-gsoc-11
          Mark Thomas made changes -
          Workflow Default workflow, editable Closed status [ 12563807 ] jira [ 12585341 ]
          Mark Thomas made changes -
          Workflow jira [ 12451852 ] Default workflow, editable Closed status [ 12563807 ]
          Michael McCandless made changes -
          Attachment CachedFilterIndexReader.java [ 12454467 ]
          Jason Rutherglen made changes -
          Attachment LUCENE-1536.patch [ 12419735 ]
          Jason Rutherglen made changes -
          Attachment LUCENE-1536.patch [ 12419694 ]
          Michael McCandless made changes -
          Fix Version/s 3.1 [ 12314025 ]
          Fix Version/s 2.9 [ 12312682 ]
          Jason Rutherglen made changes -
          Attachment LUCENE-1536.patch [ 12406046 ]
          Jason Rutherglen made changes -
          Attachment LUCENE-1536.patch [ 12405982 ]
          Michael McCandless made changes -
          Fix Version/s 2.9 [ 12312682 ]
          Michael McCandless made changes -
          Field Original Value New Value
          Attachment LUCENE-1536.patch [ 12399482 ]
          Michael McCandless created issue -

            People

            • Assignee:
              Uwe Schindler
              Reporter:
              Michael McCandless
            • Votes:
              2 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development