[LUCENE-2199] ShingleFilter skips over trie-shingles if outputUnigram is set to false

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.9, 2.9.1, 3.0
• Fix Version/s: 2.9.2, 3.0.1, 4.0-ALPHA
• Component/s: modules/analysis
• Lucene Fields: New, Patch Available
• Assignee: Robert Muir
• Reporter: Simon Willnauer
• Created: 08/Jan/10 19:58
• Updated: 16/May/11 18:15
• Resolved: 10/Jan/10 19:48

Description

Spinoff from http://lucene.markmail.org/message/uq4xdjk26yduvnpa

I noticed that if I set outputUnigrams to false it gives me the same output for
maxShingleSize=2 and maxShingleSize=3.

please divide divide this this sentence

when i set maxShingleSize to 4 output is:

please divide please divide this sentence divide this this sentence

I was expecting the output as follows with maxShingleSize=3 and
outputUnigrams=false :

please divide this divide this sentence