[LUCENE-3440] FastVectorHighlighter: IDF-weighted terms for ordered fragments

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 4.0-ALPHA, 6.0
• Component/s: modules/highlighter
• Labels: FastVectorHighlighter
• Lucene Fields: New, Patch Available
• Assignee: Koji Sekiguchi
• Reporter: Sebastian Lutze
• Created: 20/Sep/11 09:32
• Updated: 28/Aug/22 12:57
• Resolved: 12/Jun/12 14:22

Description

The FastVectorHighlighter uses for every term found in a fragment an equal weight, which causes a higher ranking for fragments with a high number of words or, in the worst case, a high number of very common words than fragments that contains all of the terms used in the original query.

This patch provides ordered fragments with IDF-weighted terms:

total weight = total weight + IDF for unique term per fragment * boost of query;

The ranking-formula should be the same, or at least similar, to that one used in org.apache.lucene.search.highlight.QueryTermScorer.

The patch is simple, but it works for us.

Some ideas:

• A better approach would be moving the whole fragments-scoring into a separate class.
• Switch scoring via parameter
• Exact phrases should be given a even better score, regardless if a phrase-query was executed or not
• edismax/dismax-parameters pf, ps and pf^boost should be observed and corresponding fragments should be ranked higher