[LUCENE-5180] ShingleFilter should make shingles from trailing holes

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.6, 6.0
• Component/s: modules/analysis
• Lucene Fields: New
• Assignee: Michael McCandless
• Reporter: Michael McCandless
• Created: 19/Aug/13 17:04
• Updated: 28/Aug/22 13:51
• Resolved: 17/Sep/13 16:29

Description

When ShingleFilter hits a hole, it uses _ as the token, e.g. bigrams for "the dog barked", if you have a StopFilter removing the, would be: "_ dog", "dog barked".

But if the input ends with a stopword, e.g. "wizard of", ShingleFilter fails to produce "wizard _" due to LUCENE-3849 ... once we fix that I think we should fix ShingleFilter to make shingles for trailing holes too ...

Issue Links

• is blocked by: LUCENE-3849 position increments should be implemented by TokenStream.end()