[SOLR-2761] FSTLookup should use long-tail like discretization instead of proportional (linear)

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 3.4
• Fix Version/s: 3.5, 3.6, 4.0-ALPHA
• Component/s: spellchecker
• Assignee: Dawid Weiss
• Reporter: David Smiley
• Created: 14/Sep/11 05:49
• Updated: 22/Jan/12 21:14
• Resolved: 18/Nov/11 11:28

Description

The Suggester's FSTLookup implementation discretizes the term frequencies into a configurable number of buckets (configurable as "weightBuckets") in order to deal with FST limitations. The mapping of a source frequency into a bucket is a proportional (i.e. linear) mapping from the minimum and maximum value. I don't think this makes sense at all given the well-known long-tail like distribution of term frequencies. As a result of this problem, I've found it necessary to increase weightBuckets substantially, like >100, to get quality suggestions.

Issue Links

• is blocked by: SOLR-2762 FSTLookup returns one less suggestion than it should when onlyMorePopular=true
• is part of: SOLR-2888 FSTSuggester refactoring: utf8 storage, external sorts (OOM prevention), code cleanups
• is related to: LUCENE-3714 add suggester that uses shortest path/wFST instead of buckets