[LUCENE-8183] HyphenationCompoundWordTokenFilter creates overlapping tokens with onlyLongestMatch enabled

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 6.6
• Component/s: modules/analysis

Environment

Configuration of the analyzer:

<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.HyphenationCompoundWordTokenFilterFactory"
        hyphenator="lang/hyph_de_DR.xml" encoding="iso-8859-1"
        dictionary="lang/wordlist_de.txt"
        onlyLongestMatch="true"/>

 

• Lucene Fields: New
• Assignee: Uwe Schindler
• Reporter: Rupert Westenthaler
• Created: 23/Feb/18 11:22
• Updated: 28/Aug/22 15:26

Description

The HyphenationCompoundWordTokenFilter creates overlapping tokens even if onlyLongestMatch is enabled. 

Example:

Dictionary: gesellschaft, schaft
Hyphenator: de_DR.xml //from Apche Offo
onlyLongestMatch: true

 

text	gesellschaft	gesellschaft	schaft
raw_bytes	[67 65 73 65 6c 6c 73 63 68 61 66 74]	[67 65 73 65 6c 6c 73 63 68 61 66 74]	[73 63 68 61 66 74]
start	0	0	0
end	12	12	12
positionLength	1	1	1
type	word	word	word
position	1	1	1

IMHO this includes 2 unexpected Tokens

1. the 2nd 'gesellschaft' as it duplicates the original token
2. the 'schaft' as it is a sub-token 'gesellschaft' that is present in the dictionary

 

Issue Links

• is related to: LUCENE-3022 DictionaryCompoundWordTokenFilter Flag onlyLongestMatch has no affect