Details
-
Improvement
-
Status: Closed
-
Minor
-
Resolution: Fixed
-
None
-
None
-
None
-
New
Description
Our current implementation of BM25 does
boost * IDF * (k1+1) * tf / (tf + norm)
As (k1+1) is a constant, it is the same for every term and doesn't modify ordering. It is often omitted and I found out that the "The Probabilistic Relevance Framework: BM25 and Beyond" paper by Robertson (BM25's author) and Zaragova even describes adding (k1+1) to the numerator as a variant whose benefit is to be more comparable with Robertson/Sparck-Jones weighting, which we don't care about.
A common variant is to add a (k1 + 1) component to the
numerator of the saturation function. This is the same for all
terms, and therefore does not affect the ranking produced.
The reason for including it was to make the final formula
more compatible with the RSJ weight used on its own
Should we remove it from BM25Similarity as well?
A side-effect that I'm interested in is that integrating other score contributions (eg. via oal.document.FeatureField) would be a bit easier to reason about. For instance a weight of 3 in FeatureField#newSaturationQuery would have a similar impact as a term whose IDF is 3 (and thus docFreq ~= 5%) rather than a term whose IDF is 3/(k1 + 1).
Attachments
Issue Links
- relates to
-
SOLR-13025 SchemaSimilarityFactory fallback to LegacyBM25Similarity
-
- Closed
-
- links to
