Lucene - Core
  1. Lucene - Core
  2. LUCENE-6201

MinShouldMatchSumScorer should advance less and score lazily

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.1, 6.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      MinShouldMatchSumScorer currently computes the score eagerly, even on documents that do not eventually match if it cannot find minShouldMatch matches on the same document.

      1. LUCENE-6201.patch
        71 kB
        Adrien Grand
      2. LUCENE-6201.patch
        56 kB
        Adrien Grand
      3. LUCENE-6201.patch
        25 kB
        Adrien Grand

        Activity

        Hide
        Adrien Grand added a comment -

        I have been working on an alternative implementation that tries to generalize how our disjunction and conjunction scorers are working. It keeps track of scorers in 3 different data-structures:

        • a linked-list of scorers that are positioned on the next potential match, called 'lead' since they are used to lead the iteration
        • a heap of scorers that are beyond the next potential match called 'head', ordered by doc ID (like DisjunctionScorer)
        • a heap of scorers that are behind the next potential match called 'tail', ordered by cost (like ConjunctionScorer, although the ConjunctionScorer case is simpler since the set of scorers does not change it can just use a sorted array). This heap has a size of at most minShouldMatch - 1, which guarantees that they can't be a match on their own (since we need at least minShouldMatch matching clauses).

        When you want to move to the next document, you first move scorers from 'lead' to 'tail'. And if it overflows, you just have to advance the least-costly scorers to 'head'. Then the next potential match is the doc ID at the top of 'head' and we need to pop 'head' from all scorers which are on this doc ID. Finally, we just have to advance the least-costly scorer from 'tail' until there is a match.

        I ran benchmarks with the current implementation using the tasks file from LUCENE-4571. Some queries are slower, other queries are faster:

                            TaskQPS baseline      StdDev   QPS patch      StdDev                Pct diff
             Low3MinShouldMatch2       41.11      (7.3%)       34.29      (3.6%)  -16.6% ( -25% -   -6%)
             Low2MinShouldMatch2       24.18      (7.4%)       21.28      (2.8%)  -12.0% ( -20% -   -1%)
             Low1MinShouldMatch2       17.92      (7.1%)       16.26      (3.1%)   -9.3% ( -18% -    0%)
             HighMinShouldMatch4       23.14      (6.3%)       21.13      (3.4%)   -8.7% ( -17% -    1%)
             HighMinShouldMatch3       17.01      (6.9%)       15.73      (2.9%)   -7.5% ( -16% -    2%)
             Low1MinShouldMatch3       23.20      (6.9%)       21.49      (3.1%)   -7.4% ( -16% -    2%)
             HighMinShouldMatch2       14.48      (6.9%)       13.63      (3.4%)   -5.8% ( -15% -    4%)
             Low4MinShouldMatch2      327.94      (2.6%)      312.58      (2.4%)   -4.7% (  -9% -    0%)
             Low2MinShouldMatch3       39.53      (7.1%)       38.77      (3.2%)   -1.9% ( -11% -    9%)
             Low4MinShouldMatch0       73.34      (3.3%)       72.17      (2.1%)   -1.6% (  -6% -    3%)
             Low2MinShouldMatch0       36.11      (2.1%)       35.62      (1.5%)   -1.4% (  -4% -    2%)
             Low1MinShouldMatch4       41.57      (6.1%)       41.01      (3.3%)   -1.4% ( -10% -    8%)
             Low3MinShouldMatch0       48.49      (2.1%)       47.90      (1.7%)   -1.2% (  -4% -    2%)
             Low3MinShouldMatch3      311.34      (8.0%)      309.54      (2.4%)   -0.6% ( -10% -   10%)
             Low1MinShouldMatch0       30.28      (2.0%)       30.14      (1.1%)   -0.5% (  -3% -    2%)
             HighMinShouldMatch0       26.09      (1.7%)       25.99      (1.1%)   -0.4% (  -3% -    2%)
                        PKLookup      322.05      (3.3%)      323.17      (3.3%)    0.3% (  -6% -    7%)
             Low2MinShouldMatch4      362.28      (5.7%)      366.96      (3.0%)    1.3% (  -6% -   10%)
             Low4MinShouldMatch4     1380.17      (6.7%)     1541.42     (11.0%)   11.7% (  -5% -   31%)
             Low3MinShouldMatch4     1299.86      (6.4%)     1506.99      (4.7%)   15.9% (   4% -   28%)
             Low4MinShouldMatch3     1060.15      (6.0%)     1233.64      (3.7%)   16.4% (   6% -   27%)
        

        This implementation is very careful about not advancing more than needed which is sometimes not the right trade-off on term queries since they are so fast. I tried to measure how many times nextDoc and advance are called for the extreme queries from this benchmark: Low3MinShouldMatch2 (-16.6%) and Low4MinShouldMatch3 (16.4%).

        Low3MinShouldMatch2 trunk patch diff
        nextDoc 3317417 2385754 -28%
        advance 2565471 3196711 +25%
        total 5882888 5582465 -5%
        Low4MinShouldMatch3 trunk patch
        nextDoc 86588 320 -99%
        advance 20644 74305 +260%
        total 107232 74625 -30%

        Overall this new impl seems to especially help on queries which have low-frequency clauses and high values of minShouldMatch where its logic helps save calls to nextDoc/advance. When it does not help save many nextDoc/advance calls like in Low3MinShouldMatch2, its constant overhead makes it slower.

        The other interesting part is that it scores lazily, so I hacked luceneutil to wrap the parsed boolean queries into constant-score queries, and this time the difference is even better since the current MinShouldMatchSumScorer always computes scores:

                            TaskQPS baseline      StdDev   QPS patch      StdDev                Pct diff
             Low1MinShouldMatch0       28.88      (1.1%)       28.43      (1.4%)   -1.5% (  -3% -    0%)
             Low2MinShouldMatch0       33.95      (1.2%)       33.51      (1.4%)   -1.3% (  -3% -    1%)
             HighMinShouldMatch0       24.83      (1.4%)       24.60      (1.5%)   -0.9% (  -3% -    2%)
             Low3MinShouldMatch0       44.95      (1.4%)       44.63      (1.4%)   -0.7% (  -3% -    2%)
             Low4MinShouldMatch0       66.11      (1.5%)       65.67      (1.6%)   -0.7% (  -3% -    2%)
                        PKLookup      321.97      (2.7%)      323.22      (2.5%)    0.4% (  -4% -    5%)
             Low4MinShouldMatch2      324.15      (3.3%)      326.37      (2.0%)    0.7% (  -4% -    6%)
             Low3MinShouldMatch3      312.13      (4.5%)      319.32      (3.0%)    2.3% (  -4% -   10%)
             Low2MinShouldMatch4      362.11      (4.0%)      371.42      (3.5%)    2.6% (  -4% -   10%)
             Low3MinShouldMatch2       40.80      (7.1%)       42.67      (4.3%)    4.6% (  -6% -   17%)
             Low1MinShouldMatch4       41.61      (7.2%)       43.63      (4.2%)    4.9% (  -6% -   17%)
             HighMinShouldMatch4       22.82      (6.9%)       24.05      (4.9%)    5.4% (  -6% -   18%)
             Low2MinShouldMatch3       39.52      (7.5%)       43.15      (4.5%)    9.2% (  -2% -   22%)
             Low1MinShouldMatch3       22.82      (7.3%)       25.56      (5.5%)   12.0% (   0% -   26%)
             Low2MinShouldMatch2       23.75      (7.3%)       27.33      (4.9%)   15.1% (   2% -   29%)
             Low4MinShouldMatch4     1291.50      (7.9%)     1492.81      (4.2%)   15.6% (   3% -   30%)
             Low4MinShouldMatch3     1013.22      (4.4%)     1193.51      (4.8%)   17.8% (   8% -   28%)
             HighMinShouldMatch3       16.65      (6.8%)       19.98      (6.1%)   20.0% (   6% -   35%)
             Low3MinShouldMatch4     1188.07     (12.2%)     1445.02      (5.5%)   21.6% (   3% -   44%)
             Low1MinShouldMatch2       17.45      (6.6%)       21.64      (6.1%)   24.1% (  10% -   39%)
             HighMinShouldMatch2       14.02      (6.2%)       18.54      (7.2%)   32.3% (  17% -   48%)
        

        But actually it does not only score lazily, it also stops advancing the tail as soon as there are minShouldMatch. For instance, I measured how many times nextDoc and advance are called depending on whether we are scoring or not with the patch.

        HighMinShouldMatch3 scoring non scoring
        nextDoc 4025282 4135163 +3%
        advance 8329393 6828807 -18%
        total 12354675 10963970 -11%
        Show
        Adrien Grand added a comment - I have been working on an alternative implementation that tries to generalize how our disjunction and conjunction scorers are working. It keeps track of scorers in 3 different data-structures: a linked-list of scorers that are positioned on the next potential match, called 'lead' since they are used to lead the iteration a heap of scorers that are beyond the next potential match called 'head', ordered by doc ID (like DisjunctionScorer) a heap of scorers that are behind the next potential match called 'tail', ordered by cost (like ConjunctionScorer, although the ConjunctionScorer case is simpler since the set of scorers does not change it can just use a sorted array). This heap has a size of at most minShouldMatch - 1 , which guarantees that they can't be a match on their own (since we need at least minShouldMatch matching clauses). When you want to move to the next document, you first move scorers from 'lead' to 'tail'. And if it overflows, you just have to advance the least-costly scorers to 'head'. Then the next potential match is the doc ID at the top of 'head' and we need to pop 'head' from all scorers which are on this doc ID. Finally, we just have to advance the least-costly scorer from 'tail' until there is a match. I ran benchmarks with the current implementation using the tasks file from LUCENE-4571 . Some queries are slower, other queries are faster: TaskQPS baseline StdDev QPS patch StdDev Pct diff Low3MinShouldMatch2 41.11 (7.3%) 34.29 (3.6%) -16.6% ( -25% - -6%) Low2MinShouldMatch2 24.18 (7.4%) 21.28 (2.8%) -12.0% ( -20% - -1%) Low1MinShouldMatch2 17.92 (7.1%) 16.26 (3.1%) -9.3% ( -18% - 0%) HighMinShouldMatch4 23.14 (6.3%) 21.13 (3.4%) -8.7% ( -17% - 1%) HighMinShouldMatch3 17.01 (6.9%) 15.73 (2.9%) -7.5% ( -16% - 2%) Low1MinShouldMatch3 23.20 (6.9%) 21.49 (3.1%) -7.4% ( -16% - 2%) HighMinShouldMatch2 14.48 (6.9%) 13.63 (3.4%) -5.8% ( -15% - 4%) Low4MinShouldMatch2 327.94 (2.6%) 312.58 (2.4%) -4.7% ( -9% - 0%) Low2MinShouldMatch3 39.53 (7.1%) 38.77 (3.2%) -1.9% ( -11% - 9%) Low4MinShouldMatch0 73.34 (3.3%) 72.17 (2.1%) -1.6% ( -6% - 3%) Low2MinShouldMatch0 36.11 (2.1%) 35.62 (1.5%) -1.4% ( -4% - 2%) Low1MinShouldMatch4 41.57 (6.1%) 41.01 (3.3%) -1.4% ( -10% - 8%) Low3MinShouldMatch0 48.49 (2.1%) 47.90 (1.7%) -1.2% ( -4% - 2%) Low3MinShouldMatch3 311.34 (8.0%) 309.54 (2.4%) -0.6% ( -10% - 10%) Low1MinShouldMatch0 30.28 (2.0%) 30.14 (1.1%) -0.5% ( -3% - 2%) HighMinShouldMatch0 26.09 (1.7%) 25.99 (1.1%) -0.4% ( -3% - 2%) PKLookup 322.05 (3.3%) 323.17 (3.3%) 0.3% ( -6% - 7%) Low2MinShouldMatch4 362.28 (5.7%) 366.96 (3.0%) 1.3% ( -6% - 10%) Low4MinShouldMatch4 1380.17 (6.7%) 1541.42 (11.0%) 11.7% ( -5% - 31%) Low3MinShouldMatch4 1299.86 (6.4%) 1506.99 (4.7%) 15.9% ( 4% - 28%) Low4MinShouldMatch3 1060.15 (6.0%) 1233.64 (3.7%) 16.4% ( 6% - 27%) This implementation is very careful about not advancing more than needed which is sometimes not the right trade-off on term queries since they are so fast. I tried to measure how many times nextDoc and advance are called for the extreme queries from this benchmark: Low3MinShouldMatch2 (-16.6%) and Low4MinShouldMatch3 (16.4%). Low3MinShouldMatch2 trunk patch diff nextDoc 3317417 2385754 -28% advance 2565471 3196711 +25% total 5882888 5582465 -5% Low4MinShouldMatch3 trunk patch nextDoc 86588 320 -99% advance 20644 74305 +260% total 107232 74625 -30% Overall this new impl seems to especially help on queries which have low-frequency clauses and high values of minShouldMatch where its logic helps save calls to nextDoc/advance. When it does not help save many nextDoc/advance calls like in Low3MinShouldMatch2, its constant overhead makes it slower. The other interesting part is that it scores lazily, so I hacked luceneutil to wrap the parsed boolean queries into constant-score queries, and this time the difference is even better since the current MinShouldMatchSumScorer always computes scores: TaskQPS baseline StdDev QPS patch StdDev Pct diff Low1MinShouldMatch0 28.88 (1.1%) 28.43 (1.4%) -1.5% ( -3% - 0%) Low2MinShouldMatch0 33.95 (1.2%) 33.51 (1.4%) -1.3% ( -3% - 1%) HighMinShouldMatch0 24.83 (1.4%) 24.60 (1.5%) -0.9% ( -3% - 2%) Low3MinShouldMatch0 44.95 (1.4%) 44.63 (1.4%) -0.7% ( -3% - 2%) Low4MinShouldMatch0 66.11 (1.5%) 65.67 (1.6%) -0.7% ( -3% - 2%) PKLookup 321.97 (2.7%) 323.22 (2.5%) 0.4% ( -4% - 5%) Low4MinShouldMatch2 324.15 (3.3%) 326.37 (2.0%) 0.7% ( -4% - 6%) Low3MinShouldMatch3 312.13 (4.5%) 319.32 (3.0%) 2.3% ( -4% - 10%) Low2MinShouldMatch4 362.11 (4.0%) 371.42 (3.5%) 2.6% ( -4% - 10%) Low3MinShouldMatch2 40.80 (7.1%) 42.67 (4.3%) 4.6% ( -6% - 17%) Low1MinShouldMatch4 41.61 (7.2%) 43.63 (4.2%) 4.9% ( -6% - 17%) HighMinShouldMatch4 22.82 (6.9%) 24.05 (4.9%) 5.4% ( -6% - 18%) Low2MinShouldMatch3 39.52 (7.5%) 43.15 (4.5%) 9.2% ( -2% - 22%) Low1MinShouldMatch3 22.82 (7.3%) 25.56 (5.5%) 12.0% ( 0% - 26%) Low2MinShouldMatch2 23.75 (7.3%) 27.33 (4.9%) 15.1% ( 2% - 29%) Low4MinShouldMatch4 1291.50 (7.9%) 1492.81 (4.2%) 15.6% ( 3% - 30%) Low4MinShouldMatch3 1013.22 (4.4%) 1193.51 (4.8%) 17.8% ( 8% - 28%) HighMinShouldMatch3 16.65 (6.8%) 19.98 (6.1%) 20.0% ( 6% - 35%) Low3MinShouldMatch4 1188.07 (12.2%) 1445.02 (5.5%) 21.6% ( 3% - 44%) Low1MinShouldMatch2 17.45 (6.6%) 21.64 (6.1%) 24.1% ( 10% - 39%) HighMinShouldMatch2 14.02 (6.2%) 18.54 (7.2%) 32.3% ( 17% - 48%) But actually it does not only score lazily, it also stops advancing the tail as soon as there are minShouldMatch . For instance, I measured how many times nextDoc and advance are called depending on whether we are scoring or not with the patch. HighMinShouldMatch3 scoring non scoring nextDoc 4025282 4135163 +3% advance 8329393 6828807 -18% total 12354675 10963970 -11%
        Hide
        Robert Muir added a comment -

        One concern i have is this seems to make the fast queries faster and the slow ones slower. The bottleneck for many apps will be the behavior on high freq queries (e.g. HighMinShouldMatch4)

        An alternative way to implement this, where there is less advance()'ing is to implement it in BS1 with the new range api.

        Show
        Robert Muir added a comment - One concern i have is this seems to make the fast queries faster and the slow ones slower. The bottleneck for many apps will be the behavior on high freq queries (e.g. HighMinShouldMatch4) An alternative way to implement this, where there is less advance()'ing is to implement it in BS1 with the new range api.
        Hide
        Adrien Grand added a comment -

        An alternative way to implement this, where there is less advance()'ing is to implement it in BS1 with the new range api.

        It's actually what I had started working on until I noticed this intriguing TODO in MinShouldMatchSumScorer that it should score lazily. I actually think we need both approaches, I'll try to see if I can merge my two patches to get global speedups on this queries.

        Show
        Adrien Grand added a comment - An alternative way to implement this, where there is less advance()'ing is to implement it in BS1 with the new range api. It's actually what I had started working on until I noticed this intriguing TODO in MinShouldMatchSumScorer that it should score lazily. I actually think we need both approaches, I'll try to see if I can merge my two patches to get global speedups on this queries.
        Hide
        Adrien Grand added a comment -

        New patch: BooleanScorer can now also deal with minShouldMatch. The way it works is that it scores all windows of 2048 documents where at least minShouldMatch clauses have a match. However, there is no guarantee about the intersection of the matches so it is only used for minShouldMatch > 1 when matches are likely dense.

        Here are results from the luceneutil benchmark:

                            TaskQPS baseline      StdDev   QPS patch      StdDev                Pct diff
             Low4MinShouldMatch4     1349.50      (6.4%)     1064.22      (3.7%)  -21.1% ( -29% -  -11%)
             Low3MinShouldMatch4     1225.14     (11.9%)      977.61      (4.6%)  -20.2% ( -32% -   -4%)
             Low4MinShouldMatch3     1040.26      (5.4%)      859.33      (3.0%)  -17.4% ( -24% -   -9%)
             Low4MinShouldMatch2      316.21      (4.6%)      281.75      (2.5%)  -10.9% ( -17% -   -3%)
             Low2MinShouldMatch4      349.07      (7.8%)      316.85      (4.8%)   -9.2% ( -20% -    3%)
             Low3MinShouldMatch3      308.45      (5.4%)      280.00      (2.2%)   -9.2% ( -15% -   -1%)
             Low4MinShouldMatch0       72.57      (2.9%)       74.43     (11.3%)    2.6% ( -11% -   17%)
             Low2MinShouldMatch3       38.11     (10.5%)       39.30     (12.6%)    3.1% ( -18% -   29%)
             Low3MinShouldMatch0       47.95      (2.4%)       49.45     (12.5%)    3.1% ( -11% -   18%)
             Low1MinShouldMatch4       39.78      (9.7%)       41.05      (2.5%)    3.2% (  -8% -   16%)
                        PKLookup      316.64      (2.5%)      327.40      (3.6%)    3.4% (  -2% -    9%)
             Low1MinShouldMatch0       30.13      (1.6%)       31.15     (12.8%)    3.4% ( -10% -   18%)
             Low2MinShouldMatch0       35.75      (1.8%)       37.01     (12.6%)    3.5% ( -10% -   18%)
             HighMinShouldMatch0       25.94      (1.4%)       26.90     (13.0%)    3.7% ( -10% -   18%)
             Low3MinShouldMatch2       39.56     (10.3%)       47.62     (13.0%)   20.4% (  -2% -   48%)
             HighMinShouldMatch4       22.28     (10.0%)       27.59     (15.5%)   23.8% (  -1% -   54%)
             Low1MinShouldMatch3       22.25     (10.5%)       31.02     (16.4%)   39.4% (  11% -   74%)
             Low2MinShouldMatch2       23.24     (10.3%)       35.63     (17.5%)   53.3% (  23% -   90%)
             HighMinShouldMatch3       16.31     (10.0%)       26.31     (19.6%)   61.3% (  28% -  101%)
             Low1MinShouldMatch2       17.24      (9.7%)       30.30     (21.2%)   75.8% (  40% -  118%)
             HighMinShouldMatch2       13.98      (9.0%)       26.28     (23.2%)   88.0% (  51% -  132%)
        

        This time we have the slow queries that become faster but also the fast queries that become slower.

        • Queries with minShouldMatch=0 seem to be faster only because BooleanScorer is used for more queries which seems to make the JVM happy (if I modify the patch to stop using BooleanScorer when minShouldMatch > 0, it's not the case anymore)
        • On the other hand queries like Low4MinShouldMatch4 are slower. I tried to revert MinShouldMatchSumScorer to the previous impl and got similar results. It seems like the fact that MinShouldMatchSumScorer is not used for most queries in this benchmark anymore make the JVM unhappy
        Show
        Adrien Grand added a comment - New patch: BooleanScorer can now also deal with minShouldMatch. The way it works is that it scores all windows of 2048 documents where at least minShouldMatch clauses have a match. However, there is no guarantee about the intersection of the matches so it is only used for minShouldMatch > 1 when matches are likely dense. Here are results from the luceneutil benchmark: TaskQPS baseline StdDev QPS patch StdDev Pct diff Low4MinShouldMatch4 1349.50 (6.4%) 1064.22 (3.7%) -21.1% ( -29% - -11%) Low3MinShouldMatch4 1225.14 (11.9%) 977.61 (4.6%) -20.2% ( -32% - -4%) Low4MinShouldMatch3 1040.26 (5.4%) 859.33 (3.0%) -17.4% ( -24% - -9%) Low4MinShouldMatch2 316.21 (4.6%) 281.75 (2.5%) -10.9% ( -17% - -3%) Low2MinShouldMatch4 349.07 (7.8%) 316.85 (4.8%) -9.2% ( -20% - 3%) Low3MinShouldMatch3 308.45 (5.4%) 280.00 (2.2%) -9.2% ( -15% - -1%) Low4MinShouldMatch0 72.57 (2.9%) 74.43 (11.3%) 2.6% ( -11% - 17%) Low2MinShouldMatch3 38.11 (10.5%) 39.30 (12.6%) 3.1% ( -18% - 29%) Low3MinShouldMatch0 47.95 (2.4%) 49.45 (12.5%) 3.1% ( -11% - 18%) Low1MinShouldMatch4 39.78 (9.7%) 41.05 (2.5%) 3.2% ( -8% - 16%) PKLookup 316.64 (2.5%) 327.40 (3.6%) 3.4% ( -2% - 9%) Low1MinShouldMatch0 30.13 (1.6%) 31.15 (12.8%) 3.4% ( -10% - 18%) Low2MinShouldMatch0 35.75 (1.8%) 37.01 (12.6%) 3.5% ( -10% - 18%) HighMinShouldMatch0 25.94 (1.4%) 26.90 (13.0%) 3.7% ( -10% - 18%) Low3MinShouldMatch2 39.56 (10.3%) 47.62 (13.0%) 20.4% ( -2% - 48%) HighMinShouldMatch4 22.28 (10.0%) 27.59 (15.5%) 23.8% ( -1% - 54%) Low1MinShouldMatch3 22.25 (10.5%) 31.02 (16.4%) 39.4% ( 11% - 74%) Low2MinShouldMatch2 23.24 (10.3%) 35.63 (17.5%) 53.3% ( 23% - 90%) HighMinShouldMatch3 16.31 (10.0%) 26.31 (19.6%) 61.3% ( 28% - 101%) Low1MinShouldMatch2 17.24 (9.7%) 30.30 (21.2%) 75.8% ( 40% - 118%) HighMinShouldMatch2 13.98 (9.0%) 26.28 (23.2%) 88.0% ( 51% - 132%) This time we have the slow queries that become faster but also the fast queries that become slower. Queries with minShouldMatch=0 seem to be faster only because BooleanScorer is used for more queries which seems to make the JVM happy (if I modify the patch to stop using BooleanScorer when minShouldMatch > 0, it's not the case anymore) On the other hand queries like Low4MinShouldMatch4 are slower. I tried to revert MinShouldMatchSumScorer to the previous impl and got similar results. It seems like the fact that MinShouldMatchSumScorer is not used for most queries in this benchmark anymore make the JVM unhappy
        Hide
        Adrien Grand added a comment -

        With this basic skipping support, maybe we can look into using bulk scorers for conjunctions too in the future...

        Show
        Adrien Grand added a comment - With this basic skipping support, maybe we can look into using bulk scorers for conjunctions too in the future...
        Hide
        Adrien Grand added a comment -

        Hmm, actually I played with it and the results are not good. I modified the patch to make pure conjunctions handled as disjunctions with minShouldMatch=number_of_optional_clauses. It is using the same heuristics as in the current patch (cost > maxDoc / 3) so only AndHighHigh runs with BooleanScorer:

                            TaskQPS baseline      StdDev   QPS patch      StdDev                Pct diff
                     AndHighHigh       85.24      (1.8%)       57.61      (8.0%)  -32.4% ( -41% -  -23%)
                          Fuzzy2       65.07     (12.3%)       59.37     (12.7%)   -8.8% ( -30% -   18%)
                          Fuzzy1       41.38      (3.4%)       39.09      (5.1%)   -5.5% ( -13% -    3%)
                      AndHighLow      781.86      (3.7%)      753.32      (3.7%)   -3.6% ( -10% -    3%)
                HighSloppyPhrase       36.01      (2.9%)       35.20      (3.5%)   -2.2% (  -8% -    4%)
                       OrHighLow       92.94      (4.4%)       91.18      (5.8%)   -1.9% ( -11% -    8%)
                 MedSloppyPhrase       64.84      (3.0%)       63.67      (3.3%)   -1.8% (  -7% -    4%)
                      AndHighMed      197.21      (2.0%)      194.06      (3.3%)   -1.6% (  -6% -    3%)
                 LowSloppyPhrase       97.46      (2.0%)       96.00      (2.2%)   -1.5% (  -5% -    2%)
                       OrHighMed       41.52      (4.8%)       41.15      (6.6%)   -0.9% ( -11% -   11%)
                      HighPhrase       35.18      (7.3%)       34.97      (7.7%)   -0.6% ( -14% -   15%)
                      OrHighHigh       25.83      (5.3%)       25.69      (6.9%)   -0.6% ( -12% -   12%)
                       MedPhrase       31.25      (3.4%)       31.17      (3.8%)   -0.3% (  -7% -    7%)
                   OrHighNotHigh       49.00      (1.9%)       48.87      (1.7%)   -0.3% (  -3% -    3%)
                         MedTerm      246.68      (2.2%)      246.06      (2.2%)   -0.3% (  -4% -    4%)
                    OrHighNotMed       40.13      (2.9%)       40.05      (2.8%)   -0.2% (  -5% -    5%)
                        HighTerm       95.31      (2.3%)       95.20      (2.2%)   -0.1% (  -4% -    4%)
                          IntNRQ        7.70      (4.7%)        7.69      (4.9%)   -0.1% (  -9% -    9%)
                   OrNotHighHigh       29.34      (1.4%)       29.32      (2.2%)   -0.1% (  -3% -    3%)
                        Wildcard       59.61      (2.9%)       59.60      (3.1%)   -0.0% (  -5% -    6%)
                     LowSpanNear      115.48      (2.6%)      115.51      (2.4%)    0.0% (  -4% -    5%)
                       LowPhrase      368.22      (3.1%)      368.37      (3.4%)    0.0% (  -6% -    6%)
                         Prefix3       97.37      (3.7%)       97.52      (3.9%)    0.2% (  -7% -    8%)
                         LowTerm      868.88      (4.8%)      870.53      (6.6%)    0.2% ( -10% -   12%)
                        PKLookup      269.02      (3.2%)      269.55      (3.4%)    0.2% (  -6% -    7%)
                    OrNotHighMed      113.70      (2.0%)      114.02      (2.5%)    0.3% (  -4% -    4%)
                     MedSpanNear      109.45      (1.8%)      109.80      (1.8%)    0.3% (  -3% -    3%)
                    OrHighNotLow      103.81      (3.1%)      104.17      (2.6%)    0.3% (  -5% -    6%)
                    HighSpanNear       10.89      (2.1%)       10.94      (1.9%)    0.4% (  -3% -    4%)
                         Respell       86.63      (6.1%)       87.09      (6.1%)    0.5% ( -10% -   13%)
                    OrNotHighLow      967.00      (4.1%)     1019.09      (3.3%)    5.4% (  -1% -   13%)
        

        BooleanScorer is not ready to run pure conjunctions. Actually this probably makes sense: one of the reason why disjunctions (with or without minShouldMatch) are slow is that they need to update some priority queues all the time in order to figure out the next candidate. BooleanScorer speeds things up by only balancing priority queues once per window. On the other hand, the ConjunctionScorer doesn't have such issues, so BooleanScorer mostly adds overhead by first collecting into a bit set and then recollecting from the bit set into the actual collector.

        Show
        Adrien Grand added a comment - Hmm, actually I played with it and the results are not good. I modified the patch to make pure conjunctions handled as disjunctions with minShouldMatch=number_of_optional_clauses. It is using the same heuristics as in the current patch (cost > maxDoc / 3) so only AndHighHigh runs with BooleanScorer: TaskQPS baseline StdDev QPS patch StdDev Pct diff AndHighHigh 85.24 (1.8%) 57.61 (8.0%) -32.4% ( -41% - -23%) Fuzzy2 65.07 (12.3%) 59.37 (12.7%) -8.8% ( -30% - 18%) Fuzzy1 41.38 (3.4%) 39.09 (5.1%) -5.5% ( -13% - 3%) AndHighLow 781.86 (3.7%) 753.32 (3.7%) -3.6% ( -10% - 3%) HighSloppyPhrase 36.01 (2.9%) 35.20 (3.5%) -2.2% ( -8% - 4%) OrHighLow 92.94 (4.4%) 91.18 (5.8%) -1.9% ( -11% - 8%) MedSloppyPhrase 64.84 (3.0%) 63.67 (3.3%) -1.8% ( -7% - 4%) AndHighMed 197.21 (2.0%) 194.06 (3.3%) -1.6% ( -6% - 3%) LowSloppyPhrase 97.46 (2.0%) 96.00 (2.2%) -1.5% ( -5% - 2%) OrHighMed 41.52 (4.8%) 41.15 (6.6%) -0.9% ( -11% - 11%) HighPhrase 35.18 (7.3%) 34.97 (7.7%) -0.6% ( -14% - 15%) OrHighHigh 25.83 (5.3%) 25.69 (6.9%) -0.6% ( -12% - 12%) MedPhrase 31.25 (3.4%) 31.17 (3.8%) -0.3% ( -7% - 7%) OrHighNotHigh 49.00 (1.9%) 48.87 (1.7%) -0.3% ( -3% - 3%) MedTerm 246.68 (2.2%) 246.06 (2.2%) -0.3% ( -4% - 4%) OrHighNotMed 40.13 (2.9%) 40.05 (2.8%) -0.2% ( -5% - 5%) HighTerm 95.31 (2.3%) 95.20 (2.2%) -0.1% ( -4% - 4%) IntNRQ 7.70 (4.7%) 7.69 (4.9%) -0.1% ( -9% - 9%) OrNotHighHigh 29.34 (1.4%) 29.32 (2.2%) -0.1% ( -3% - 3%) Wildcard 59.61 (2.9%) 59.60 (3.1%) -0.0% ( -5% - 6%) LowSpanNear 115.48 (2.6%) 115.51 (2.4%) 0.0% ( -4% - 5%) LowPhrase 368.22 (3.1%) 368.37 (3.4%) 0.0% ( -6% - 6%) Prefix3 97.37 (3.7%) 97.52 (3.9%) 0.2% ( -7% - 8%) LowTerm 868.88 (4.8%) 870.53 (6.6%) 0.2% ( -10% - 12%) PKLookup 269.02 (3.2%) 269.55 (3.4%) 0.2% ( -6% - 7%) OrNotHighMed 113.70 (2.0%) 114.02 (2.5%) 0.3% ( -4% - 4%) MedSpanNear 109.45 (1.8%) 109.80 (1.8%) 0.3% ( -3% - 3%) OrHighNotLow 103.81 (3.1%) 104.17 (2.6%) 0.3% ( -5% - 6%) HighSpanNear 10.89 (2.1%) 10.94 (1.9%) 0.4% ( -3% - 4%) Respell 86.63 (6.1%) 87.09 (6.1%) 0.5% ( -10% - 13%) OrNotHighLow 967.00 (4.1%) 1019.09 (3.3%) 5.4% ( -1% - 13%) BooleanScorer is not ready to run pure conjunctions. Actually this probably makes sense: one of the reason why disjunctions (with or without minShouldMatch) are slow is that they need to update some priority queues all the time in order to figure out the next candidate. BooleanScorer speeds things up by only balancing priority queues once per window. On the other hand, the ConjunctionScorer doesn't have such issues, so BooleanScorer mostly adds overhead by first collecting into a bit set and then recollecting from the bit set into the actual collector.
        Hide
        Adrien Grand added a comment -

        Here is a new patch and a summary of what it is doing:

        • improve MinShouldMatchSumScorer to call nextDoc/advance less on sub scorers
        • improve MinShouldMatchSumScorer to only score on demand
        • make BooleanScorer able to deal with minShouldMatch > 1. The way it works is that it only scores windows of 2048 documents where at least minShouldMatch clauses have at least one match.
        • make BooleanScorer used for minShouldMatch > 1 when the cost is high (> maxDoc / 3)
        • DisjunctionScorer and MinShouldMatchSumScorer both had a priority queue ordered by doc ID so I factored it out to a separate class (but not using oal.util.PriorityQueue since the pluggable comparison function seems to hurt performance a lot)

        Here are results of the various benchmarks that we have:

        wikimedium10M:

                            TaskQPS baseline      StdDev   QPS patch      StdDev                Pct diff
                          Fuzzy2       58.35      (4.4%)       55.53      (6.4%)   -4.8% ( -14% -    6%)
                       OrHighMed       59.68      (4.0%)       59.11      (4.2%)   -1.0% (  -8% -    7%)
                       OrHighLow       68.54      (4.0%)       67.91      (4.3%)   -0.9% (  -8% -    7%)
                      OrHighHigh       21.65      (4.6%)       21.48      (4.8%)   -0.8% (  -9% -    9%)
                          Fuzzy1       80.72      (3.8%)       80.40      (3.6%)   -0.4% (  -7% -    7%)
                        HighTerm      105.98      (2.5%)      105.59      (2.8%)   -0.4% (  -5% -    5%)
                        Wildcard       49.40      (5.3%)       49.24      (5.3%)   -0.3% ( -10% -   10%)
                         MedTerm      220.04      (2.6%)      219.36      (2.8%)   -0.3% (  -5% -    5%)
                    OrHighNotLow       92.87      (2.8%)       92.61      (2.8%)   -0.3% (  -5% -    5%)
                    OrHighNotMed       67.10      (2.3%)       66.92      (2.2%)   -0.3% (  -4% -    4%)
                         Prefix3      100.43      (5.4%)      100.19      (5.3%)   -0.2% ( -10% -   11%)
                        PKLookup      272.56      (3.2%)      271.93      (3.3%)   -0.2% (  -6% -    6%)
                    OrNotHighMed      171.29      (2.1%)      170.92      (2.1%)   -0.2% (  -4% -    4%)
                         LowTerm      822.27      (5.0%)      821.03      (4.8%)   -0.2% (  -9% -   10%)
                       MedPhrase      137.90      (2.5%)      137.77      (2.2%)   -0.1% (  -4% -    4%)
                   OrNotHighHigh       42.48      (1.4%)       42.44      (1.3%)   -0.1% (  -2% -    2%)
                     AndHighHigh       52.45      (1.3%)       52.43      (1.3%)   -0.0% (  -2% -    2%)
                      AndHighMed      210.77      (2.1%)      210.83      (2.4%)    0.0% (  -4% -    4%)
                       LowPhrase       50.86      (3.8%)       50.90      (3.7%)    0.1% (  -7% -    7%)
                     MedSpanNear       19.83      (5.6%)       19.84      (5.6%)    0.1% ( -10% -   11%)
                      HighPhrase       18.50      (3.3%)       18.52      (3.4%)    0.1% (  -6% -    7%)
                 LowSloppyPhrase       41.13      (2.6%)       41.17      (2.2%)    0.1% (  -4% -    5%)
                 MedSloppyPhrase       60.91      (3.0%)       60.98      (2.4%)    0.1% (  -5% -    5%)
                   OrHighNotHigh       40.01      (1.2%)       40.06      (1.3%)    0.1% (  -2% -    2%)
                          IntNRQ       16.17      (6.2%)       16.19      (6.3%)    0.1% ( -11% -   13%)
                    OrNotHighLow      611.94      (3.1%)      612.86      (3.3%)    0.1% (  -6% -    6%)
                    HighSpanNear        3.47      (5.8%)        3.48      (6.1%)    0.2% ( -11% -   12%)
                HighSloppyPhrase       24.65      (3.8%)       24.71      (3.2%)    0.3% (  -6% -    7%)
                     LowSpanNear       90.25      (2.8%)       90.61      (3.1%)    0.4% (  -5% -    6%)
                         Respell       72.37      (3.9%)       72.95      (3.7%)    0.8% (  -6% -    8%)
                      AndHighLow      821.42      (4.4%)      830.72      (4.0%)    1.1% (  -6% -    9%)
        

        wikimedium10M with lucene (both the baseline and the patched version) patched to forcefully use BS2 all the time (to validate that sharing the pq between DisjunctionScorer and MinShouldMatchSumScorer does not hurt):

                            TaskQPS baseline      StdDev   QPS patch      StdDev                Pct diff
                          Fuzzy1       72.12      (5.7%)       71.59      (7.2%)   -0.7% ( -12% -   12%)
                         LowTerm      860.24      (5.5%)      855.31      (5.1%)   -0.6% ( -10% -   10%)
                        PKLookup      267.21      (2.6%)      266.07      (2.7%)   -0.4% (  -5% -    4%)
                    OrNotHighMed      244.11      (1.6%)      243.11      (2.0%)   -0.4% (  -3% -    3%)
                       MedPhrase      226.75      (2.3%)      226.07      (2.3%)   -0.3% (  -4% -    4%)
                   OrHighNotHigh       47.84      (1.6%)       47.71      (1.8%)   -0.3% (  -3% -    3%)
                HighSloppyPhrase       24.65      (2.4%)       24.59      (2.4%)   -0.2% (  -4% -    4%)
                    OrHighNotMed       53.28      (2.9%)       53.15      (3.1%)   -0.2% (  -6% -    5%)
                        HighTerm       54.46      (2.3%)       54.36      (2.3%)   -0.2% (  -4% -    4%)
                      HighPhrase       46.80      (2.6%)       46.73      (2.7%)   -0.2% (  -5% -    5%)
                       LowPhrase      224.46      (3.1%)      224.16      (3.6%)   -0.1% (  -6% -    6%)
                    OrHighNotLow       75.21      (3.3%)       75.11      (3.2%)   -0.1% (  -6% -    6%)
                   OrNotHighHigh       55.94      (1.5%)       55.91      (1.8%)   -0.1% (  -3% -    3%)
                    OrNotHighLow      932.36      (3.4%)      932.28      (3.2%)   -0.0% (  -6% -    6%)
                         MedTerm      306.76      (2.3%)      306.77      (2.3%)    0.0% (  -4% -    4%)
                     AndHighHigh       59.85      (0.9%)       59.87      (0.9%)    0.0% (  -1% -    1%)
                         Prefix3       29.55      (2.4%)       29.56      (2.5%)    0.0% (  -4% -    5%)
                 LowSloppyPhrase       35.97      (3.0%)       36.01      (2.7%)    0.1% (  -5% -    5%)
                     LowSpanNear      219.56      (3.0%)      219.86      (3.2%)    0.1% (  -5% -    6%)
                       OrHighLow       17.18      (3.2%)       17.21      (4.2%)    0.2% (  -7% -    7%)
                          IntNRQ        8.77      (2.8%)        8.79      (3.0%)    0.2% (  -5% -    6%)
                      AndHighMed      184.88      (1.5%)      185.24      (1.5%)    0.2% (  -2% -    3%)
                        Wildcard       40.48      (2.5%)       40.56      (2.6%)    0.2% (  -4% -    5%)
                 MedSloppyPhrase       35.40      (2.4%)       35.47      (2.1%)    0.2% (  -4% -    4%)
                    HighSpanNear        7.32      (4.8%)        7.35      (5.1%)    0.5% (  -8% -   10%)
                         Respell       58.99      (4.0%)       59.35      (2.7%)    0.6% (  -5% -    7%)
                      AndHighLow      921.16      (6.8%)      927.79      (4.1%)    0.7% (  -9% -   12%)
                     MedSpanNear       10.33      (3.3%)       10.41      (3.2%)    0.7% (  -5% -    7%)
                          Fuzzy2       64.92     (11.8%)       65.56     (10.1%)    1.0% ( -18% -   25%)
                       OrHighMed       38.65      (2.8%)       39.25      (4.4%)    1.6% (  -5% -    9%)
                      OrHighHigh       20.06      (2.7%)       20.64      (4.7%)    2.9% (  -4% -   10%)
        

        MinShouldMatch tasks on wikimedium1M:

             Low3MinShouldMatch4     1204.57      (7.1%)      944.59      (2.7%)  -21.6% ( -29% -  -12%)
             Low4MinShouldMatch4     1294.39     (10.7%)     1026.20      (3.8%)  -20.7% ( -31% -   -6%)
             Low4MinShouldMatch3      993.94      (7.7%)      827.22      (5.4%)  -16.8% ( -27% -   -3%)
             Low4MinShouldMatch2      314.76      (4.9%)      272.76      (3.4%)  -13.3% ( -20% -   -5%)
             Low2MinShouldMatch4      345.04      (8.0%)      303.89      (6.4%)  -11.9% ( -24% -    2%)
             Low3MinShouldMatch3      304.41      (5.0%)      271.79      (3.3%)  -10.7% ( -18% -   -2%)
             Low1MinShouldMatch4       39.30      (8.8%)       39.70      (2.7%)    1.0% (  -9% -   13%)
             Low4MinShouldMatch0       71.65      (3.2%)       73.38      (6.5%)    2.4% (  -7% -   12%)
             Low3MinShouldMatch0       47.45      (2.2%)       49.00      (7.2%)    3.3% (  -5% -   12%)
             Low2MinShouldMatch3       37.50      (9.3%)       38.83      (8.4%)    3.5% ( -12% -   23%)
             HighMinShouldMatch0       25.55      (1.8%)       26.52      (8.1%)    3.8% (  -6% -   13%)
             Low2MinShouldMatch0       35.24      (1.8%)       36.61      (7.3%)    3.9% (  -5% -   13%)
                        PKLookup      316.26      (2.5%)      328.56      (3.4%)    3.9% (  -1% -   10%)
             Low1MinShouldMatch0       29.63      (2.1%)       30.89      (7.6%)    4.2% (  -5% -   14%)
             Low3MinShouldMatch2       38.91      (9.6%)       47.22      (8.4%)   21.3% (   3% -   43%)
             HighMinShouldMatch4       21.97      (9.4%)       27.15     (10.2%)   23.5% (   3% -   47%)
             Low1MinShouldMatch3       21.84      (9.5%)       30.69     (10.9%)   40.5% (  18% -   67%)
             Low2MinShouldMatch2       22.69      (9.7%)       35.21     (11.0%)   55.2% (  31% -   83%)
             HighMinShouldMatch3       16.01      (9.7%)       26.07     (12.3%)   62.9% (  37% -   94%)
             Low1MinShouldMatch2       16.86      (9.3%)       29.94     (13.1%)   77.6% (  50% -  110%)
             HighMinShouldMatch2       13.64      (8.6%)       25.99     (14.6%)   90.6% (  62% -  124%)
        

        On this last benchmark, we can see that the use of BooleanScorer helps slow queries. Some queries are slower but it looks like it's just because MinShouldMatchSumScorer gets picked less often.

        I think it's ready?

        Show
        Adrien Grand added a comment - Here is a new patch and a summary of what it is doing: improve MinShouldMatchSumScorer to call nextDoc/advance less on sub scorers improve MinShouldMatchSumScorer to only score on demand make BooleanScorer able to deal with minShouldMatch > 1. The way it works is that it only scores windows of 2048 documents where at least minShouldMatch clauses have at least one match. make BooleanScorer used for minShouldMatch > 1 when the cost is high (> maxDoc / 3) DisjunctionScorer and MinShouldMatchSumScorer both had a priority queue ordered by doc ID so I factored it out to a separate class (but not using oal.util.PriorityQueue since the pluggable comparison function seems to hurt performance a lot) Here are results of the various benchmarks that we have: wikimedium10M: TaskQPS baseline StdDev QPS patch StdDev Pct diff Fuzzy2 58.35 (4.4%) 55.53 (6.4%) -4.8% ( -14% - 6%) OrHighMed 59.68 (4.0%) 59.11 (4.2%) -1.0% ( -8% - 7%) OrHighLow 68.54 (4.0%) 67.91 (4.3%) -0.9% ( -8% - 7%) OrHighHigh 21.65 (4.6%) 21.48 (4.8%) -0.8% ( -9% - 9%) Fuzzy1 80.72 (3.8%) 80.40 (3.6%) -0.4% ( -7% - 7%) HighTerm 105.98 (2.5%) 105.59 (2.8%) -0.4% ( -5% - 5%) Wildcard 49.40 (5.3%) 49.24 (5.3%) -0.3% ( -10% - 10%) MedTerm 220.04 (2.6%) 219.36 (2.8%) -0.3% ( -5% - 5%) OrHighNotLow 92.87 (2.8%) 92.61 (2.8%) -0.3% ( -5% - 5%) OrHighNotMed 67.10 (2.3%) 66.92 (2.2%) -0.3% ( -4% - 4%) Prefix3 100.43 (5.4%) 100.19 (5.3%) -0.2% ( -10% - 11%) PKLookup 272.56 (3.2%) 271.93 (3.3%) -0.2% ( -6% - 6%) OrNotHighMed 171.29 (2.1%) 170.92 (2.1%) -0.2% ( -4% - 4%) LowTerm 822.27 (5.0%) 821.03 (4.8%) -0.2% ( -9% - 10%) MedPhrase 137.90 (2.5%) 137.77 (2.2%) -0.1% ( -4% - 4%) OrNotHighHigh 42.48 (1.4%) 42.44 (1.3%) -0.1% ( -2% - 2%) AndHighHigh 52.45 (1.3%) 52.43 (1.3%) -0.0% ( -2% - 2%) AndHighMed 210.77 (2.1%) 210.83 (2.4%) 0.0% ( -4% - 4%) LowPhrase 50.86 (3.8%) 50.90 (3.7%) 0.1% ( -7% - 7%) MedSpanNear 19.83 (5.6%) 19.84 (5.6%) 0.1% ( -10% - 11%) HighPhrase 18.50 (3.3%) 18.52 (3.4%) 0.1% ( -6% - 7%) LowSloppyPhrase 41.13 (2.6%) 41.17 (2.2%) 0.1% ( -4% - 5%) MedSloppyPhrase 60.91 (3.0%) 60.98 (2.4%) 0.1% ( -5% - 5%) OrHighNotHigh 40.01 (1.2%) 40.06 (1.3%) 0.1% ( -2% - 2%) IntNRQ 16.17 (6.2%) 16.19 (6.3%) 0.1% ( -11% - 13%) OrNotHighLow 611.94 (3.1%) 612.86 (3.3%) 0.1% ( -6% - 6%) HighSpanNear 3.47 (5.8%) 3.48 (6.1%) 0.2% ( -11% - 12%) HighSloppyPhrase 24.65 (3.8%) 24.71 (3.2%) 0.3% ( -6% - 7%) LowSpanNear 90.25 (2.8%) 90.61 (3.1%) 0.4% ( -5% - 6%) Respell 72.37 (3.9%) 72.95 (3.7%) 0.8% ( -6% - 8%) AndHighLow 821.42 (4.4%) 830.72 (4.0%) 1.1% ( -6% - 9%) wikimedium10M with lucene (both the baseline and the patched version) patched to forcefully use BS2 all the time (to validate that sharing the pq between DisjunctionScorer and MinShouldMatchSumScorer does not hurt): TaskQPS baseline StdDev QPS patch StdDev Pct diff Fuzzy1 72.12 (5.7%) 71.59 (7.2%) -0.7% ( -12% - 12%) LowTerm 860.24 (5.5%) 855.31 (5.1%) -0.6% ( -10% - 10%) PKLookup 267.21 (2.6%) 266.07 (2.7%) -0.4% ( -5% - 4%) OrNotHighMed 244.11 (1.6%) 243.11 (2.0%) -0.4% ( -3% - 3%) MedPhrase 226.75 (2.3%) 226.07 (2.3%) -0.3% ( -4% - 4%) OrHighNotHigh 47.84 (1.6%) 47.71 (1.8%) -0.3% ( -3% - 3%) HighSloppyPhrase 24.65 (2.4%) 24.59 (2.4%) -0.2% ( -4% - 4%) OrHighNotMed 53.28 (2.9%) 53.15 (3.1%) -0.2% ( -6% - 5%) HighTerm 54.46 (2.3%) 54.36 (2.3%) -0.2% ( -4% - 4%) HighPhrase 46.80 (2.6%) 46.73 (2.7%) -0.2% ( -5% - 5%) LowPhrase 224.46 (3.1%) 224.16 (3.6%) -0.1% ( -6% - 6%) OrHighNotLow 75.21 (3.3%) 75.11 (3.2%) -0.1% ( -6% - 6%) OrNotHighHigh 55.94 (1.5%) 55.91 (1.8%) -0.1% ( -3% - 3%) OrNotHighLow 932.36 (3.4%) 932.28 (3.2%) -0.0% ( -6% - 6%) MedTerm 306.76 (2.3%) 306.77 (2.3%) 0.0% ( -4% - 4%) AndHighHigh 59.85 (0.9%) 59.87 (0.9%) 0.0% ( -1% - 1%) Prefix3 29.55 (2.4%) 29.56 (2.5%) 0.0% ( -4% - 5%) LowSloppyPhrase 35.97 (3.0%) 36.01 (2.7%) 0.1% ( -5% - 5%) LowSpanNear 219.56 (3.0%) 219.86 (3.2%) 0.1% ( -5% - 6%) OrHighLow 17.18 (3.2%) 17.21 (4.2%) 0.2% ( -7% - 7%) IntNRQ 8.77 (2.8%) 8.79 (3.0%) 0.2% ( -5% - 6%) AndHighMed 184.88 (1.5%) 185.24 (1.5%) 0.2% ( -2% - 3%) Wildcard 40.48 (2.5%) 40.56 (2.6%) 0.2% ( -4% - 5%) MedSloppyPhrase 35.40 (2.4%) 35.47 (2.1%) 0.2% ( -4% - 4%) HighSpanNear 7.32 (4.8%) 7.35 (5.1%) 0.5% ( -8% - 10%) Respell 58.99 (4.0%) 59.35 (2.7%) 0.6% ( -5% - 7%) AndHighLow 921.16 (6.8%) 927.79 (4.1%) 0.7% ( -9% - 12%) MedSpanNear 10.33 (3.3%) 10.41 (3.2%) 0.7% ( -5% - 7%) Fuzzy2 64.92 (11.8%) 65.56 (10.1%) 1.0% ( -18% - 25%) OrHighMed 38.65 (2.8%) 39.25 (4.4%) 1.6% ( -5% - 9%) OrHighHigh 20.06 (2.7%) 20.64 (4.7%) 2.9% ( -4% - 10%) MinShouldMatch tasks on wikimedium1M: Low3MinShouldMatch4 1204.57 (7.1%) 944.59 (2.7%) -21.6% ( -29% - -12%) Low4MinShouldMatch4 1294.39 (10.7%) 1026.20 (3.8%) -20.7% ( -31% - -6%) Low4MinShouldMatch3 993.94 (7.7%) 827.22 (5.4%) -16.8% ( -27% - -3%) Low4MinShouldMatch2 314.76 (4.9%) 272.76 (3.4%) -13.3% ( -20% - -5%) Low2MinShouldMatch4 345.04 (8.0%) 303.89 (6.4%) -11.9% ( -24% - 2%) Low3MinShouldMatch3 304.41 (5.0%) 271.79 (3.3%) -10.7% ( -18% - -2%) Low1MinShouldMatch4 39.30 (8.8%) 39.70 (2.7%) 1.0% ( -9% - 13%) Low4MinShouldMatch0 71.65 (3.2%) 73.38 (6.5%) 2.4% ( -7% - 12%) Low3MinShouldMatch0 47.45 (2.2%) 49.00 (7.2%) 3.3% ( -5% - 12%) Low2MinShouldMatch3 37.50 (9.3%) 38.83 (8.4%) 3.5% ( -12% - 23%) HighMinShouldMatch0 25.55 (1.8%) 26.52 (8.1%) 3.8% ( -6% - 13%) Low2MinShouldMatch0 35.24 (1.8%) 36.61 (7.3%) 3.9% ( -5% - 13%) PKLookup 316.26 (2.5%) 328.56 (3.4%) 3.9% ( -1% - 10%) Low1MinShouldMatch0 29.63 (2.1%) 30.89 (7.6%) 4.2% ( -5% - 14%) Low3MinShouldMatch2 38.91 (9.6%) 47.22 (8.4%) 21.3% ( 3% - 43%) HighMinShouldMatch4 21.97 (9.4%) 27.15 (10.2%) 23.5% ( 3% - 47%) Low1MinShouldMatch3 21.84 (9.5%) 30.69 (10.9%) 40.5% ( 18% - 67%) Low2MinShouldMatch2 22.69 (9.7%) 35.21 (11.0%) 55.2% ( 31% - 83%) HighMinShouldMatch3 16.01 (9.7%) 26.07 (12.3%) 62.9% ( 37% - 94%) Low1MinShouldMatch2 16.86 (9.3%) 29.94 (13.1%) 77.6% ( 50% - 110%) HighMinShouldMatch2 13.64 (8.6%) 25.99 (14.6%) 90.6% ( 62% - 124%) On this last benchmark, we can see that the use of BooleanScorer helps slow queries. Some queries are slower but it looks like it's just because MinShouldMatchSumScorer gets picked less often. I think it's ready?
        Hide
        Adrien Grand added a comment -

        I forgot to mention I also added cost() support to BulkScorer which is useful to run boolean queries with minShouldMatch > 1 efficiently with BooleanScorer.

        Show
        Adrien Grand added a comment - I forgot to mention I also added cost() support to BulkScorer which is useful to run boolean queries with minShouldMatch > 1 efficiently with BooleanScorer.
        Hide
        ASF subversion and git services added a comment -

        Commit 1657286 from Adrien Grand in branch 'dev/trunk'
        [ https://svn.apache.org/r1657286 ]

        LUCENE-6201: Improvements to minShouldMatch handling.

        Show
        ASF subversion and git services added a comment - Commit 1657286 from Adrien Grand in branch 'dev/trunk' [ https://svn.apache.org/r1657286 ] LUCENE-6201 : Improvements to minShouldMatch handling.
        Hide
        ASF subversion and git services added a comment -

        Commit 1657290 from Adrien Grand in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1657290 ]

        LUCENE-6201: Improvements to minShouldMatch handling.

        Show
        ASF subversion and git services added a comment - Commit 1657290 from Adrien Grand in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1657290 ] LUCENE-6201 : Improvements to minShouldMatch handling.
        Hide
        Timothy Potter added a comment -

        Bulk close after 5.1 release

        Show
        Timothy Potter added a comment - Bulk close after 5.1 release

          People

          • Assignee:
            Adrien Grand
            Reporter:
            Adrien Grand
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development