[LUCENE-3417] DictionaryCompoundWordTokenFilter does not properly add tokens from the end compound word.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3, 4.0-ALPHA
• Fix Version/s: 3.5, 4.0-ALPHA
• Component/s: modules/analysis
• Lucene Fields: New, Patch Available
• Assignee: Robert Muir
• Reporter: Njal Karevoll
• Created: 06/Sep/11 20:05
• Updated: 28/Nov/24 18:02
• Resolved: 08/Sep/11 15:18

Description

Due to an off-by-one error, a subword placed at the end of a compound word will not get a token added to the token stream.

For example (from the unit test in the attached patch):
Dictionary:

{"ab", "cd", "ef"}

Input: "abcdef"
Created tokens:

{"abcdef", "ab", "cd"}

Expected tokens:

{"abcdef", "ab", "cd", "ef"}

Additionally, it could produce tokens that were shorter than the minSubwordSize due to another off-by-one error. For example (again, from the attached patch):

Dictionary:

{"abc", "d", "efg"}

Minimum subword length: 2
Input: "abcdefg"
Created tokens:

{"abcdef", "abc", "d", "efg"}

Expected tokens:

{"abcdef", "abc", "efg"}