Lucene - Core
  1. Lucene - Core
  2. LUCENE-3212

Supply FilterIndexReader based on any o.a.l.search.Filter

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: 4.0-ALPHA
    • Fix Version/s: 4.0-ALPHA
    • Component/s: core/index, core/search
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      When coding LUCENE-2919 (PKIndexSplitter), Mike and me had the idea, how to effectively apply filters on the lowest level (before query execution). This is very useful for e.g. security Filters that simply hide some documents. Currently when you apply the filter after searching, lots of useless work was done like scoring filtered documents, iterating term positions (for Phrases),...

      This patch will provide a FilterIndexReader subclass (4.0 only, 3.x is too complicated to implement), that hides filtered documents by returning them in getDeletedDocs(). In contrast to LUCENE-2919, the filtering will work on per-segment (without SlowMultiReaderWrapper), so per segment search keeps available and reopening can be done very efficient, as the filter is only calculated on openeing new or changed segments.

      This filter should improve use-cases where the filter can be applied one time before all queries (like security filters) on (re-)opening the IndexReader.

        Issue Links

          Activity

          Show
          Michael McCandless added a comment - I think this is idea is similar to the CachedFilterIndexReader on LUCENE-1536 ? See https://issues.apache.org/jira/browse/LUCENE-1536?focusedCommentId=12908914&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12908914
          Hide
          Uwe Schindler added a comment -

          It's similar, but I dont understand the impl there. I would simply override getDeletedDocs to return the deleted docs ored with the filtered. Then you dont need to override terms() and fields().

          Show
          Uwe Schindler added a comment - It's similar, but I dont understand the impl there. I would simply override getDeletedDocs to return the deleted docs ored with the filtered. Then you dont need to override terms() and fields().
          Hide
          Michael McCandless added a comment -

          That's a good point – I'm not sure why I didn't just override getDeletedDocs! It seems like that should work fine.

          Show
          Michael McCandless added a comment - That's a good point – I'm not sure why I didn't just override getDeletedDocs! It seems like that should work fine.
          Hide
          Uwe Schindler added a comment -

          I don't think, this issue is obsolete with LUCENE-1536:
          If you have one filter thats e.g. applied for one user every time, maybe for all his queries, it can live as long as the SegmentReader lives. So simply wrapping the IndexReader with a Filter has much more flexibility, as its done one time on creating the IndexReader - so I think, this filter could additionally live in contrib. If we have RandomAccessFilters, this one and also PKIndexSplitter (which will only use this FIR and drop its own impl) can directly use the Bits supplied by the Filter's DocIdSet.

          Show
          Uwe Schindler added a comment - I don't think, this issue is obsolete with LUCENE-1536 : If you have one filter thats e.g. applied for one user every time, maybe for all his queries, it can live as long as the SegmentReader lives. So simply wrapping the IndexReader with a Filter has much more flexibility, as its done one time on creating the IndexReader - so I think, this filter could additionally live in contrib. If we have RandomAccessFilters, this one and also PKIndexSplitter (which will only use this FIR and drop its own impl) can directly use the Bits supplied by the Filter's DocIdSet.
          Hide
          Uwe Schindler added a comment -

          LUCENE-1536 superceedes this.

          Show
          Uwe Schindler added a comment - LUCENE-1536 superceedes this.

            People

            • Assignee:
              Uwe Schindler
              Reporter:
              Uwe Schindler
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development