[LUCENE-4955] NGramTokenFilter increments positions for each gram

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.3
• Fix Version/s: 4.4, 6.0
• Component/s: modules/analysis
• Lucene Fields: New
• Assignee: Adrien Grand
• Reporter: Simon Willnauer
• Created: 24/Apr/13 16:40
• Updated: 28/Aug/22 13:44
• Resolved: 26/Apr/13 14:34

Description

NGramTokenFilter increments positions for each gram rather for the actual token which can lead to rather funny problems especially with highlighting. if this filter should be used for highlighting is a different story but today this seems to be a common practice in many situations to highlight sub-term matches.

I have a test for highlighting that uses ngram failing with a StringIOOB since tokens are sorted by position which causes offsets to be mixed up due to ngram token filter.

Issue Links

• is related to: LUCENE-3920 ngram tokenizer/filters create nonsense offsets if followed by a word combiner
• relates to: LUCENE-3920 ngram tokenizer/filters create nonsense offsets if followed by a word combiner