[LUCENE-8501] An ability to define the sum method for custom term frequencies

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Component/s: core/index
• Lucene Fields: New
• Assignee: Unassigned
• Reporter: Olli Kuonanoja
• Created: 17/Sep/18 07:55
• Updated: 28/Aug/22 15:35
• Resolved: 19/Sep/18 13:47

Description

Custom term frequencies allows expert users to index and score in custom ways, however, DefaultIndexingChain adds a limitation to this as the sum of frequencies can't overflow

try {
    invertState.length = Math.addExact(invertState.length, invertState.termFreqAttribute.getTermFrequency());
} catch (ArithmeticException ae) {
    throw new IllegalArgumentException("too many tokens for field \"" + field.name() + "\"");
}

This might become an issue if for example the frequency data is encoded in a different way, say the specific scorer works with float frequencies.

The sum method can be added to TermFrequencyAttribute to get something like

invertState.length = invertState.termFreqAttribute.addFrequency(invertState.length);

so users may define the summing method and avoid the owerflow exceptions.