Lucene - Core
  1. Lucene - Core
  2. LUCENE-6601

Change FilteredQuery.FilterStrategy to use the two-phase iteration API

    Details

    • Type: Task Task
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.3
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      We could change FilterStrategy so that instead of being a factory of scorers, it would just rewrite filters in such a way that they can decide which of the iterator or random-access API should be used.

      1. LUCENE-6601.patch
        35 kB
        Adrien Grand

        Activity

        Hide
        Adrien Grand added a comment -

        Here is a patch against 5.x:

        • FilteredQuery always rewrites to a BooleanQuery
        • FilterStrategy was changed to have a single method that rewrites the filter to a wrapper that can decide to either use the iterator API or the random-access API to implement the scorer
        • QUERY_FIRST_FILTER_STRATEGY and LEAP_FROG_QUERY_FIRST_STRATEGY are now the same thing and use the iterator API, which one is iterated first depends on the cost API
        • QUERY_FIRST_FILTER_STRATEGY applies the filter as an approximation
        • RANDOM_ACCESS_FILTER_STRATEGY applies cheap filters using the iterator API. However a change is that costly filters are now applied as an approximation instead of acceptDocs. If the query is a simple term query, this doesn't make any difference, but in case of more sophisticated queries, Lucene will now execute the "cheap" components of the query before the filter.

        This way FilteredQuery does not use acceptDocs anymore, which means we could backport LUCENE-6553 to 5.x.

        Show
        Adrien Grand added a comment - Here is a patch against 5.x: FilteredQuery always rewrites to a BooleanQuery FilterStrategy was changed to have a single method that rewrites the filter to a wrapper that can decide to either use the iterator API or the random-access API to implement the scorer QUERY_FIRST_FILTER_STRATEGY and LEAP_FROG_QUERY_FIRST_STRATEGY are now the same thing and use the iterator API, which one is iterated first depends on the cost API QUERY_FIRST_FILTER_STRATEGY applies the filter as an approximation RANDOM_ACCESS_FILTER_STRATEGY applies cheap filters using the iterator API. However a change is that costly filters are now applied as an approximation instead of acceptDocs. If the query is a simple term query, this doesn't make any difference, but in case of more sophisticated queries, Lucene will now execute the "cheap" components of the query before the filter. This way FilteredQuery does not use acceptDocs anymore, which means we could backport LUCENE-6553 to 5.x.
        Hide
        ASF subversion and git services added a comment -

        Commit 1687514 from Adrien Grand in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1687514 ]

        LUCENE-6601: Make FilteredQuery.FilterStrategy leverage the two-phase iteration API instead of the acceptDocs parameter of the Scorer API.

        Show
        ASF subversion and git services added a comment - Commit 1687514 from Adrien Grand in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1687514 ] LUCENE-6601 : Make FilteredQuery.FilterStrategy leverage the two-phase iteration API instead of the acceptDocs parameter of the Scorer API.
        Hide
        ASF subversion and git services added a comment -

        Commit 1687515 from Adrien Grand in branch 'dev/trunk'
        [ https://svn.apache.org/r1687515 ]

        LUCENE-6601: Backport CHANGES entries to trunk.

        Show
        ASF subversion and git services added a comment - Commit 1687515 from Adrien Grand in branch 'dev/trunk' [ https://svn.apache.org/r1687515 ] LUCENE-6601 : Backport CHANGES entries to trunk.
        Hide
        Joel Bernstein added a comment -

        I believe this ticket caused this test failure:

        Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Linux/13026/
        Java: 32bit/jdk1.9.0-ea-b60 -server -XX:+UseSerialGC

        1 tests failed.
        FAILED: org.apache.solr.search.TestHashQParserPlugin.testHashPartition

        Error Message:
        Query does not implement createWeight

        Stack Trace:
        java.lang.UnsupportedOperationException: Query does not implement createWeight
        at __randomizedtesting.SeedInfo.seed([ABA75E60C74B5463:7A1A7F85FC21ED08]:0)
        at org.apache.lucene.search.Query.createWeight(Query.java:79)
        at org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:851)
        at org.apache.lucene.search.ConstantScoreQuery.createWeight(ConstantScoreQuery.java:117)
        at org.apache.solr.search.HashQParserPlugin$HashQuery.createWeight(HashQParserPlugin.java:151)

        ant test -Dtestcase=TestHashQParserPlugin -Dtests.method=testHashPartition -Dtests.seed=3CDD0F5BDDF9D8B3 -Dtests.slow=true -Dtests.locale=iw_IL -Dtests.timezone=Pacific/Johnston -Dtests.asserts=true -Dtests.file.encoding=UTF-8

        Show
        Joel Bernstein added a comment - I believe this ticket caused this test failure: Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Linux/13026/ Java: 32bit/jdk1.9.0-ea-b60 -server -XX:+UseSerialGC 1 tests failed. FAILED: org.apache.solr.search.TestHashQParserPlugin.testHashPartition Error Message: Query does not implement createWeight Stack Trace: java.lang.UnsupportedOperationException: Query does not implement createWeight at __randomizedtesting.SeedInfo.seed( [ABA75E60C74B5463:7A1A7F85FC21ED08] :0) at org.apache.lucene.search.Query.createWeight(Query.java:79) at org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:851) at org.apache.lucene.search.ConstantScoreQuery.createWeight(ConstantScoreQuery.java:117) at org.apache.solr.search.HashQParserPlugin$HashQuery.createWeight(HashQParserPlugin.java:151) ant test -Dtestcase=TestHashQParserPlugin -Dtests.method=testHashPartition -Dtests.seed=3CDD0F5BDDF9D8B3 -Dtests.slow=true -Dtests.locale=iw_IL -Dtests.timezone=Pacific/Johnston -Dtests.asserts=true -Dtests.file.encoding=UTF-8
        Hide
        Joel Bernstein added a comment -

        The failure is due to the removal of the createWeight method from the Filter class.

        I'll need to find a different way to implement:
        at org.apache.solr.search.HashQParserPlugin$HashQuery.createWeight(HashQParserPlugin.java:151)

        Show
        Joel Bernstein added a comment - The failure is due to the removal of the createWeight method from the Filter class. I'll need to find a different way to implement: at org.apache.solr.search.HashQParserPlugin$HashQuery.createWeight(HashQParserPlugin.java:151)
        Hide
        Adrien Grand added a comment -

        The contract of the Query API is that you first need to call rewrite as long as rewrite returns a different query (see IndexSearcher.rewrite), and this will give you a Query that supports createWeight. So in this particular case, I think we just need to make sure to call rewrite first? I already committed such a change and it makes the test pass, please let me know if it somehow changes the behaviour of this HashQParser in an undesired way?

        Show
        Adrien Grand added a comment - The contract of the Query API is that you first need to call rewrite as long as rewrite returns a different query (see IndexSearcher.rewrite), and this will give you a Query that supports createWeight. So in this particular case, I think we just need to make sure to call rewrite first? I already committed such a change and it makes the test pass, please let me know if it somehow changes the behaviour of this HashQParser in an undesired way?
        Hide
        Joel Bernstein added a comment -

        Ok, thanks for the explanation and fix. I'll make sure to call rewrite before createWeight in the future.

        Show
        Joel Bernstein added a comment - Ok, thanks for the explanation and fix. I'll make sure to call rewrite before createWeight in the future.
        Hide
        Shalin Shekhar Mangar added a comment -

        Bulk close for 5.3.0 release

        Show
        Shalin Shekhar Mangar added a comment - Bulk close for 5.3.0 release

          People

          • Assignee:
            Adrien Grand
            Reporter:
            Adrien Grand
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development