Lucene - Core
  1. Lucene - Core
  2. LUCENE-6251

Two-phase support on ConstantScorer, ReqOptScorer, ReqExclScorer, BoostedScorer

    Details

    • Type: Task Task
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.1, 6.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Some of our scorers are mostly delegators and two-phase support should be reasonably easy to implement.

      1. LUCENE-6251.patch
        18 kB
        Adrien Grand

        Activity

        Hide
        Adrien Grand added a comment - - edited

        Here is a patch. I think the most interesting scorer in the list is ReqExclScorer since it can take advantage of the fact that its req and/or excl sub scorers support approximations.

        luceneutil still seems happy (the Or*Not* tasks are relevant here):

                            TaskQPS baseline      StdDev   QPS patch      StdDev                Pct diff
                          Fuzzy1       92.93      (6.5%)       90.94      (5.8%)   -2.1% ( -13% -   10%)
                         LowTerm      877.65      (4.1%)      859.71      (3.1%)   -2.0% (  -8% -    5%)
                      AndHighLow      785.85      (2.2%)      779.92      (1.9%)   -0.8% (  -4% -    3%)
                    OrNotHighLow      599.56      (4.0%)      595.90      (3.8%)   -0.6% (  -8% -    7%)
                        HighTerm      118.40      (3.2%)      117.74      (3.2%)   -0.6% (  -6% -    5%)
                         Respell       55.63      (3.8%)       55.32      (4.3%)   -0.6% (  -8% -    7%)
                 LowSloppyPhrase       78.47      (3.6%)       78.13      (3.7%)   -0.4% (  -7% -    7%)
                HighSloppyPhrase       12.38      (3.0%)       12.34      (3.1%)   -0.3% (  -6% -    6%)
                 MedSloppyPhrase       36.66      (2.7%)       36.57      (2.6%)   -0.2% (  -5% -    5%)
                       OrHighMed       55.16      (8.7%)       55.07      (8.8%)   -0.2% ( -16% -   18%)
                     MedSpanNear       20.57      (1.7%)       20.54      (1.6%)   -0.2% (  -3% -    3%)
                    OrHighNotLow       36.52      (2.3%)       36.46      (1.2%)   -0.1% (  -3% -    3%)
                      OrHighHigh       40.06      (9.1%)       40.03      (9.0%)   -0.1% ( -16% -   19%)
                         MedTerm      309.26      (2.9%)      308.98      (3.0%)   -0.1% (  -5% -    6%)
                       LowPhrase       35.25      (1.2%)       35.24      (1.1%)   -0.0% (  -2% -    2%)
                     LowSpanNear       18.45      (1.4%)       18.44      (1.3%)   -0.0% (  -2% -    2%)
                        PKLookup      274.81      (2.5%)      274.76      (2.8%)   -0.0% (  -5% -    5%)
                       OrHighLow       17.74      (9.3%)       17.74      (9.5%)   -0.0% ( -17% -   20%)
                        Wildcard       86.99      (3.6%)       87.00      (3.6%)    0.0% (  -6% -    7%)
                      AndHighMed       87.13      (1.8%)       87.17      (1.7%)    0.0% (  -3% -    3%)
                       MedPhrase       94.52      (2.9%)       94.57      (3.0%)    0.1% (  -5% -    6%)
                    HighSpanNear       11.63      (1.4%)       11.64      (1.4%)    0.1% (  -2% -    2%)
                      HighPhrase       21.24      (1.8%)       21.31      (1.8%)    0.3% (  -3% -    3%)
                     AndHighHigh       52.22      (1.9%)       52.38      (1.9%)    0.3% (  -3% -    4%)
                    OrNotHighMed      175.12      (2.3%)      175.83      (2.1%)    0.4% (  -3% -    4%)
                         Prefix3       32.36      (7.1%)       32.50      (6.7%)    0.5% ( -12% -   15%)
                          IntNRQ       16.21      (6.6%)       16.34      (6.1%)    0.8% ( -11% -   14%)
                   OrNotHighHigh       48.00      (1.4%)       48.44      (1.7%)    0.9% (  -2% -    4%)
                    OrHighNotMed       64.52      (2.1%)       65.15      (1.0%)    1.0% (  -2% -    4%)
                          Fuzzy2       55.07     (17.8%)       56.37     (18.2%)    2.4% ( -28% -   46%)
                   OrHighNotHigh       50.10      (1.5%)       51.40      (1.7%)    2.6% (   0% -    5%)
        
        Show
        Adrien Grand added a comment - - edited Here is a patch. I think the most interesting scorer in the list is ReqExclScorer since it can take advantage of the fact that its req and/or excl sub scorers support approximations. luceneutil still seems happy (the Or*Not* tasks are relevant here): TaskQPS baseline StdDev QPS patch StdDev Pct diff Fuzzy1 92.93 (6.5%) 90.94 (5.8%) -2.1% ( -13% - 10%) LowTerm 877.65 (4.1%) 859.71 (3.1%) -2.0% ( -8% - 5%) AndHighLow 785.85 (2.2%) 779.92 (1.9%) -0.8% ( -4% - 3%) OrNotHighLow 599.56 (4.0%) 595.90 (3.8%) -0.6% ( -8% - 7%) HighTerm 118.40 (3.2%) 117.74 (3.2%) -0.6% ( -6% - 5%) Respell 55.63 (3.8%) 55.32 (4.3%) -0.6% ( -8% - 7%) LowSloppyPhrase 78.47 (3.6%) 78.13 (3.7%) -0.4% ( -7% - 7%) HighSloppyPhrase 12.38 (3.0%) 12.34 (3.1%) -0.3% ( -6% - 6%) MedSloppyPhrase 36.66 (2.7%) 36.57 (2.6%) -0.2% ( -5% - 5%) OrHighMed 55.16 (8.7%) 55.07 (8.8%) -0.2% ( -16% - 18%) MedSpanNear 20.57 (1.7%) 20.54 (1.6%) -0.2% ( -3% - 3%) OrHighNotLow 36.52 (2.3%) 36.46 (1.2%) -0.1% ( -3% - 3%) OrHighHigh 40.06 (9.1%) 40.03 (9.0%) -0.1% ( -16% - 19%) MedTerm 309.26 (2.9%) 308.98 (3.0%) -0.1% ( -5% - 6%) LowPhrase 35.25 (1.2%) 35.24 (1.1%) -0.0% ( -2% - 2%) LowSpanNear 18.45 (1.4%) 18.44 (1.3%) -0.0% ( -2% - 2%) PKLookup 274.81 (2.5%) 274.76 (2.8%) -0.0% ( -5% - 5%) OrHighLow 17.74 (9.3%) 17.74 (9.5%) -0.0% ( -17% - 20%) Wildcard 86.99 (3.6%) 87.00 (3.6%) 0.0% ( -6% - 7%) AndHighMed 87.13 (1.8%) 87.17 (1.7%) 0.0% ( -3% - 3%) MedPhrase 94.52 (2.9%) 94.57 (3.0%) 0.1% ( -5% - 6%) HighSpanNear 11.63 (1.4%) 11.64 (1.4%) 0.1% ( -2% - 2%) HighPhrase 21.24 (1.8%) 21.31 (1.8%) 0.3% ( -3% - 3%) AndHighHigh 52.22 (1.9%) 52.38 (1.9%) 0.3% ( -3% - 4%) OrNotHighMed 175.12 (2.3%) 175.83 (2.1%) 0.4% ( -3% - 4%) Prefix3 32.36 (7.1%) 32.50 (6.7%) 0.5% ( -12% - 15%) IntNRQ 16.21 (6.6%) 16.34 (6.1%) 0.8% ( -11% - 14%) OrNotHighHigh 48.00 (1.4%) 48.44 (1.7%) 0.9% ( -2% - 4%) OrHighNotMed 64.52 (2.1%) 65.15 (1.0%) 1.0% ( -2% - 4%) Fuzzy2 55.07 (17.8%) 56.37 (18.2%) 2.4% ( -28% - 46%) OrHighNotHigh 50.10 (1.5%) 51.40 (1.7%) 2.6% ( 0% - 5%)
        Hide
        Adrien Grand added a comment -

        With this patch applied, the only boolean queries that would not either take advantage (conjunctions/exclusions) or propagate (delegators/disjunctions) approximations are those that have a minShouldMatch > 1.

        Show
        Adrien Grand added a comment - With this patch applied, the only boolean queries that would not either take advantage (conjunctions/exclusions) or propagate (delegators/disjunctions) approximations are those that have a minShouldMatch > 1.
        Hide
        Robert Muir added a comment -

        +1 especially for the added tests

        Show
        Robert Muir added a comment - +1 especially for the added tests
        Hide
        ASF subversion and git services added a comment -

        Commit 1660318 from Adrien Grand in branch 'dev/trunk'
        [ https://svn.apache.org/r1660318 ]

        LUCENE-6251: Add two-phase support to ConstantScorer and most boolean scorers.

        Show
        ASF subversion and git services added a comment - Commit 1660318 from Adrien Grand in branch 'dev/trunk' [ https://svn.apache.org/r1660318 ] LUCENE-6251 : Add two-phase support to ConstantScorer and most boolean scorers.
        Hide
        ASF subversion and git services added a comment -

        Commit 1660325 from Adrien Grand in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1660325 ]

        LUCENE-6251: Add two-phase support to ConstantScorer and most boolean scorers.

        Show
        ASF subversion and git services added a comment - Commit 1660325 from Adrien Grand in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1660325 ] LUCENE-6251 : Add two-phase support to ConstantScorer and most boolean scorers.
        Hide
        Timothy Potter added a comment -

        Bulk close after 5.1 release

        Show
        Timothy Potter added a comment - Bulk close after 5.1 release

          People

          • Assignee:
            Adrien Grand
            Reporter:
            Adrien Grand
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development