[LUCENE-6818] Implementing Divergence from Independence (DFI) Term-Weighting for Lucene/Solr

• Type: New Feature
• Status: Reopened
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 5.3
• Fix Version/s: 5.5, 6.0
• Component/s: core/query/scoring
• Labels: similarity
• Lucene Fields: New, Patch Available
• Assignee: Robert Muir
• Reporter: Ahmet Arslan
• Created: 27/Sep/15 15:57
• Updated: 19/Sep/24 09:45
• Resolved: 18/Jan/16 09:51

Description

As explained in the write-up, many state-of-the-art ranking model implementations are added to Apache Lucene.

This issue aims to include DFI model, which is the non-parametric counterpart of the Divergence from Randomness (DFR) framework.

DFI is both parameter-free and non-parametric:

• parameter-free: it does not require any parameter tuning or training.
• non-parametric: it does not make any assumptions about word frequency distributions on document collections.

It is highly recommended not to remove stopwords (very common terms: the, of, and, to, a, in, for, is, on, that, etc) with this similarity.

For more information see: A nonparametric term weighting method for information retrieval based on measuring the divergence from independence

Issue Links

• relates to: LUCENE-2959 [GSoC] Implementing State of the Art Ranking for Lucene