[LUCENE-6986] Add more DFI independence measures

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 5.5, 6.0
• Lucene Fields: New
• Assignee: Unassigned
• Reporter: Robert Muir
• Created: 21/Jan/16 05:43
• Updated: 28/Aug/22 14:48
• Resolved: 22/Jan/16 13:55

Description

Since LUCENE-6818 we have DFISimilarity which implements normalized chi-squared distance.

But there are other alternatives (as described in http://trec.nist.gov/pubs/trec21/papers/irra.web.nb.pdf):

• normalized chi-squared: "can be used for tasks that require high precision, against both short and long queries"
• standardized: "good at tasks that require high recall and high precision, especially against short queries composed of a few words as in the case of Internet searches"
• saturated: "for tasks that require high recall against long queries"

I think we should just provide the three independence measures, and let the user choose. Similar to how we do DFR/IB/etc.