[LUCENE-3038] DictionaryCompoundWordTokenFilter fails to create some tokens for final parts of words

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 3.1, 4.0-ALPHA
• Fix Version/s: 3.1, 4.0-ALPHA
• Component/s: modules/analysis
• Lucene Fields: New, Patch Available
• Assignee: Unassigned
• Reporter: Filip Svendsen
• Created: 19/Apr/11 21:46
• Updated: 28/Aug/22 12:44
• Resolved: 19/Apr/11 23:03

Description

DictionaryCompoundWordTokenFilter: Due to an off-by-one error, a word component placed last in a compound word, will not get a token if its length is equal to the minimal sub-word length.

Example:
min sub-word length: 4
Dictionary:

{"alfa", "beta"}

word: "alfabeta"
Created tokens:

{"alfabeta", "alfa"}

Expected tokens:

{"alfabeta", "alfa", "beta"}

I have a patch with a testcase that fails on versions 3.1 and 4.0 (probably for everything between as well, and for previous versions), along with a bugfix.