Lucene - Core
  1. Lucene - Core
  2. LUCENE-4327

Use BooleanScorer1 for filter-down-low queries

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.0, 6.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New
    1. LUCENE-4327.patch
      4 kB
      Robert Muir
    2. LUCENE-4327.patch
      4 kB
      Robert Muir
    3. LUCENE-4327.patch
      3 kB
      Robert Muir

      Activity

      Hide
      Robert Muir added a comment -

      Problem is that BS1 and BS2 sometimes scored differently, so we couldnt safely use BS1 for any segments (otherwise scoring would be not comparable for the query).

      But when filtering down low, I think BS1 is the way to go.

      Show
      Robert Muir added a comment - Problem is that BS1 and BS2 sometimes scored differently, so we couldnt safely use BS1 for any segments (otherwise scoring would be not comparable for the query). But when filtering down low, I think BS1 is the way to go.
      Hide
      Michael McCandless added a comment -

      This results in enormous performance improvements on 10M Wiki index:

                      Task    QPS base StdDev baseQPS bs1filterStdDev bs1filter      Pct diff
                  PKLookup      215.57        1.61      207.44        4.92   -6% -    0%
                 OrHighLow       24.23        2.26       38.61        2.37   36% -   86%
                 OrHighMed       13.22        1.17       21.16        1.24   38% -   85%
                OrHighHigh        4.49        0.40        7.50        0.40   45% -   93%
      

      +1

      Show
      Michael McCandless added a comment - This results in enormous performance improvements on 10M Wiki index: Task QPS base StdDev baseQPS bs1filterStdDev bs1filter Pct diff PKLookup 215.57 1.61 207.44 4.92 -6% - 0% OrHighLow 24.23 2.26 38.61 2.37 36% - 86% OrHighMed 13.22 1.17 21.16 1.24 38% - 85% OrHighHigh 4.49 0.40 7.50 0.40 45% - 93% +1
      Hide
      Michael McCandless added a comment -

      (That was with a random "keeps 10%" filter).

      Show
      Michael McCandless added a comment - (That was with a random "keeps 10%" filter).
      Hide
      Robert Muir added a comment -

      Just adding two comments to the conjunction scorers, if in the future we send required clauses to BS1 (today we do not), we need to make the same modification so that scores are comparable with BS1/BS2 there.

      Show
      Robert Muir added a comment - Just adding two comments to the conjunction scorers, if in the future we send required clauses to BS1 (today we do not), we need to make the same modification so that scores are comparable with BS1/BS2 there.
      Hide
      Robert Muir added a comment -

      Same comment belongs in ReqOptSumScorer too.

      Show
      Robert Muir added a comment - Same comment belongs in ReqOptSumScorer too.
      Hide
      Robert Muir added a comment -

      Thanks for benchmarking Mike!

      Show
      Robert Muir added a comment - Thanks for benchmarking Mike!
      Hide
      Uwe Schindler added a comment -

      Thank you very much! Great success!

      Show
      Uwe Schindler added a comment - Thank you very much! Great success!
      Hide
      Tony Stevenson added a comment -

      Faux comment, please ignore. JIRA testing.

      Show
      Tony Stevenson added a comment - Faux comment, please ignore. JIRA testing.
      Hide
      Uwe Schindler added a comment -

      But it is text only now!

      Show
      Uwe Schindler added a comment - But it is text only now!
      Hide
      Uwe Schindler added a comment -

      Closed after release.

      Show
      Uwe Schindler added a comment - Closed after release.

        People

        • Assignee:
          Unassigned
          Reporter:
          Robert Muir
        • Votes:
          0 Vote for this issue
          Watchers:
          2 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development