[SOLR-2887] FSTSuggester shouldn't OOM on large inputs

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Fix Version/s: 3.5, 4.0-ALPHA
• Component/s: spellchecker
• Assignee: Dawid Weiss
• Reporter: Dawid Weiss
• Created: 10/Nov/11 16:35
• Updated: 27/Nov/11 12:36
• Resolved: 18/Nov/11 11:28

Description

Currently the input to FSTSuggester needs to be re-sorted and this is done in-memory. Kind of defeats the purpose of the component since everything else is super-efficient but we don't even get to that part because of OOMs during construction.

Robert suggested using a spill-to-disk and merge sort on-disk. I suggested creating a lucene index and then enumerating terms for automaton construction or taking the automaton directly from the index structure (if it isn't pruned).

Issue Links

• is part of: SOLR-2888 FSTSuggester refactoring: utf8 storage, external sorts (OOM prevention), code cleanups