[LUCENE-9687] Hunspell support improvements

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 9.0, 8.9
• Lucene Fields: New
• Assignee: Unassigned
• Reporter: Peter Gromov
• Created: 21/Jan/21 11:57
• Updated: 28/Aug/22 16:13
• Resolved: 15/Mar/21 16:45

Description

I'd like Lucene's Hunspell support to be on a par with the native C++ Hunspell for spellchecking and suggestions, at least for some languages. So I propose to:

• support the affix rules necessary for English, German, French, Spanish and
  Russian dictionaries, possibly more languages later
• mirror Hunspell's suggestion algorithm in Lucene
• provide a public APIs for spellchecking, suggestion, stemming, morphological data
• check corpora for specific languages to find and fix spellchecking/suggestion discrepancices between Lucene's implementation and Hunspell/C++

Issue Links

• links to: GitHub Pull Request #2451
• links to: GitHub Pull Request #2499

Sub-Tasks

• Hunspell: support COMPOUNDRULE: Closed
• Hunspell: support default encoding: Resolved
• Hunspell: prefix condition is only checked on suffix, not stem: Closed
• Hunspell spellchecker: support numbers with separators: Resolved
• Hunspell: deduplicate decodeFlags+hasFlag checks: Closed
• Hunspell: fix off-by-one error to support prefixes of word.length - 1: Closed
• Hunspell: simplify Dictionary.affixData storage: Closed
• Hunspell: improve stemming of all-caps words: Closed
• Hunspell: shorten Stemmer.applyAffix: Resolved
• Hunspell: add a spellchecker, support BREAK and FORBIDDENWORD affix rules: Resolved
• Hunspell support: fix most IntelliJ warnings, cleanup: Resolved
• Hunspell: consider prefix's continuation flags when applying suffix: Resolved
• Hunspell: support special title-case for words with apostrophe: Closed
• Hunspell: support trailing comments on aff option lines: Closed
• Hunspell: extract Stemmer.stripAffix from similar code in prefix/suffix processing: Closed
• Hunspell: check that all flags are > 0 and fit char: Closed
• Hunspell Stemmer: use the same FST.BytesReader on all recursion levels: Closed
• Hunspell: reuse char[] when possible when stripping affix: Closed
• Support German-like compound words: Closed
• Hunspell: support words with trailing dots: Closed
• Hunspell: implement simple REP-based suggestion algorithm: Closed
• Hunspell: support alternate casing for short language codes: Closed
• Hunspell: prohibit FORBIDDENWORD words and their case variations: Closed
• Hunspell: support capitalization for German ß: Closed
• Hunspell: support NEEDAFFIX flag on affixes: Closed
• Hunspell: check Lucene's implementation against Hunspell's test data: Closed
• Hunspell: support FLAG UTF-8 in absence of SET UTF-8: Closed
• Hunspell: no special dotted i treatment outside tr/az languages: Closed
• Hunspell: support minor compounding-related flags: Closed
• Hunspell: support flag usage before its format is even specified: Closed
• Hunspell: support CHECKCOMPOUNDPATTERN: Closed
• Hunspell: more ways to vary misspelled word variations for suggestions: Closed
• Hunspell: disallow ONLYINCOMPOUND suffixes at the very end of compound words: Closed
• Hunspell: update sanity tests that load all dictionaries: Closed
• Hunspell: tolerate existing aff/dic file typos: Closed
• Hunspell: speed up spellchecking by stopping at a single found stem: Closed
• Add build-side support for running full validation checks against hunspell repos: Closed
• Hunspell: add a performance test: Closed
• Hunspell: support CHECKCOMPOUNDREP flags: Closed
• Clean up temporary folder management in Dictionary: Closed
• Hunspell: support dictionary entries starting with slash: Closed
• Hunspell: exception when loading dictionaries with mixed-case words and aliased flags: Closed
• Hunspell: support suggestions based on "ph" morphological data: Closed
• Hunspell: speed up flag checks by avoiding allocations: Closed
• Hunspell: support MAP-based suggestions for groups of similar letters: Closed
• Hunspell: speed up numeric flag parsing: Closed
• Avoid buffering and double-scan of flags in *.aff file: Closed
• Hunspell: suggest dictionary entries similar to the misspelled word: Closed
• Hunspell: ignore original tests which are out of scope for now: Closed
• Hunspell: tolerate more aff/dic file typos: Closed
• Hunspell: unify case variation logic in Stemmer and SpellChecker: Closed
• Hunspell: suggest inflected dictionary entries similar to the misspelled word: Closed
• Hunspell: apply output conversion (OCONV) to the suggestions: Closed
• Hunspell: improve suggestions for mixed-case misspelled words: Closed
• Hunspell Stemmer: reduce parameter count: Closed
• Hunspell: disallow compounds with parts present in dictionary space-separated: Closed
• Hunspell: support NOSUGGEST option: Closed
• Hunspell: add more to TestHunspellRepositoryTestCases.EXPECTED_FAILURES: Closed
• Hunspell: print total memory usage in TestAllDictionaries, cleanup: Closed
• Hunspell: check that FLAG and SET don't occur too far in the file, cleanup: Closed
• Hunspell: fix FORBIDDENWORD support: Closed
• Hunspell: try title case as FORCEUCASE misspelled word suggestions: Closed
• Hunspell: rename SpellChecker to Hunspell, fix test name, update javadoc and CHANGES.txt: Closed
• Hunspell: add API for retrieving dictionary morphological data and stemming: Closed
• Hunspell: KEEPCASE should take precedence over affixed forms: Closed
• Hunspell: don't perform compound check recursively when looking for space-separated word pairs: Closed
• Hunspell: don't lookup word roots unnecessarily to check flags: Closed
• Hunspell: CHECKCOMPOUNDCASE shouldn't prohibit dash-separated uppercase compounds: Closed
• Hunspell: make FORCEUCASE work when the first compound word is inherently title-case: Closed
• Hunspell: allow to inflect the last part of COMPOUNDRULE compound: Closed
• Hunspell: speed up input conversion: Closed
• Hunspell: add an API to interrupt long computations: Closed
• Hunspell suggestions: split by space (but not dash) also before last char: Closed
• Hunspell: don't suggest more than 4 ngram corrections by default: Closed
• Hunspell suggestions: use US keyboard in absence of KEY option: Closed
• Hunspell: don't check case in compound middle and end: Closed
• Hunspell suggestions: try moving the last character into the middle: Closed
• Hunspell: speed up suggesting a bit by not creating a huge TreeSet: Closed
• Hunspell: avoid slow dictionary lookup if the word's hash isn't there: Closed
• Add automation for running regression tests: Closed
• Hunspell: don't check second-level affixes when the first level isn't a continuation: Closed
• Hunspell: put a time limit on suggestion calculation: Closed
• Hunspell suggestions: speed up expandWord by enumerating only applicable affixes: Closed
• Hunspell: don't check second stage suffixes if the first stage flag only occurs in prefixes: Closed
• Hunspell: fix most similar dictionary entry search by reversing the comparator: Closed
• Hunspell: fix space + mixed case heuristics on suggestions: Closed
• Hunspell: speed up affix condition checking: Closed
• Hunspell suggestions: consider space/dash-separated words for each case variation: Closed
• Hunspell: when generating suggestions, skip too deep word FST subtrees: Closed
• Hunspell suggestions: speed up ngram calculation by not searching for substrings in impossible places: Closed
• Hunspell: honor empty stripping affixes when generating suggestions: Closed
• Hunspell suggestions: speed up ngram score calculation for each dictionary entry: Closed
• Hunspell: reverse the "words" trie for faster word lookup/suggestions: Resolved
• Hunspell: store word length for faster dictionary lookup/enumeration: Closed
• Hunspell GeneratingSuggester: faster flag & case checks, less allocations: Closed
• Hunspell: SIOOBE in GeneratingSuggester.expandRoot: Closed
• Hunspell: AssertionError in WordStorage.lookupWord: Closed
• Hunspell suggestions: speed up for some non-Latin scripts: Closed