Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-8197

Make top-k queries fast when static scoring signals are incorporated into the score

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 7.4, 8.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Block-max WAND (LUCENE-8135) and some earlier issues made Lucene faster at computing the top-k matches of boolean queries.

      It is quite frequent that users want to improve ranking and end up scoring with a formula that could look like bm25_score + w * log(alpha + pagerank) (w and alpha being constants, and pagerank being a per-document field value). You could do this with doc values and FunctionScoreQuery but unfortunately this will remove the ability to optimize top-k queries since the scoring formula becomes opaque to Lucene.

      I'd like to add a new field that allows to store such scoring signals as term frequencies, and new queries that could produce log(alpha + pagerank) as a score. Then implementing the above formula can be done by boosting this query with a boost equal to w and adding this boosted query as a SHOULD clause of a BooleanQuery. This would give Lucene the ability to compute top-k hits faster, especially but not only if the index is sorted by decreasing pagerank.

        Attachments

        1. LUCENE-8197.patch
          38 kB
          Adrien Grand
        2. LUCENE-8197.patch
          43 kB
          Adrien Grand
        3. LUCENE-8197.patch
          29 kB
          Adrien Grand
        4. LUCENE-8197.patch
          26 kB
          Adrien Grand

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              jpountz Adrien Grand
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: