Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-4327

Use BooleanScorer1 for filter-down-low queries

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.0, 6.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New
    1. LUCENE-4327.patch
      4 kB
      Robert Muir
    2. LUCENE-4327.patch
      4 kB
      Robert Muir
    3. LUCENE-4327.patch
      3 kB
      Robert Muir

      Activity

      Hide
      rcmuir Robert Muir added a comment -

      Problem is that BS1 and BS2 sometimes scored differently, so we couldnt safely use BS1 for any segments (otherwise scoring would be not comparable for the query).

      But when filtering down low, I think BS1 is the way to go.

      Show
      rcmuir Robert Muir added a comment - Problem is that BS1 and BS2 sometimes scored differently, so we couldnt safely use BS1 for any segments (otherwise scoring would be not comparable for the query). But when filtering down low, I think BS1 is the way to go.
      Hide
      mikemccand Michael McCandless added a comment -

      This results in enormous performance improvements on 10M Wiki index:

                      Task    QPS base StdDev baseQPS bs1filterStdDev bs1filter      Pct diff
                  PKLookup      215.57        1.61      207.44        4.92   -6% -    0%
                 OrHighLow       24.23        2.26       38.61        2.37   36% -   86%
                 OrHighMed       13.22        1.17       21.16        1.24   38% -   85%
                OrHighHigh        4.49        0.40        7.50        0.40   45% -   93%
      

      +1

      Show
      mikemccand Michael McCandless added a comment - This results in enormous performance improvements on 10M Wiki index: Task QPS base StdDev baseQPS bs1filterStdDev bs1filter Pct diff PKLookup 215.57 1.61 207.44 4.92 -6% - 0% OrHighLow 24.23 2.26 38.61 2.37 36% - 86% OrHighMed 13.22 1.17 21.16 1.24 38% - 85% OrHighHigh 4.49 0.40 7.50 0.40 45% - 93% +1
      Hide
      mikemccand Michael McCandless added a comment -

      (That was with a random "keeps 10%" filter).

      Show
      mikemccand Michael McCandless added a comment - (That was with a random "keeps 10%" filter).
      Hide
      rcmuir Robert Muir added a comment -

      Just adding two comments to the conjunction scorers, if in the future we send required clauses to BS1 (today we do not), we need to make the same modification so that scores are comparable with BS1/BS2 there.

      Show
      rcmuir Robert Muir added a comment - Just adding two comments to the conjunction scorers, if in the future we send required clauses to BS1 (today we do not), we need to make the same modification so that scores are comparable with BS1/BS2 there.
      Hide
      rcmuir Robert Muir added a comment -

      Same comment belongs in ReqOptSumScorer too.

      Show
      rcmuir Robert Muir added a comment - Same comment belongs in ReqOptSumScorer too.
      Hide
      rcmuir Robert Muir added a comment -

      Thanks for benchmarking Mike!

      Show
      rcmuir Robert Muir added a comment - Thanks for benchmarking Mike!
      Hide
      thetaphi Uwe Schindler added a comment -

      Thank you very much! Great success!

      Show
      thetaphi Uwe Schindler added a comment - Thank you very much! Great success!
      Hide
      pctony Tony Stevenson added a comment -

      Faux comment, please ignore. JIRA testing.

      Show
      pctony Tony Stevenson added a comment - Faux comment, please ignore. JIRA testing.
      Hide
      thetaphi Uwe Schindler added a comment -

      But it is text only now!

      Show
      thetaphi Uwe Schindler added a comment - But it is text only now!
      Hide
      thetaphi Uwe Schindler added a comment -

      Closed after release.

      Show
      thetaphi Uwe Schindler added a comment - Closed after release.

        People

        • Assignee:
          Unassigned
          Reporter:
          rcmuir Robert Muir
        • Votes:
          0 Vote for this issue
          Watchers:
          2 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development