[LUCENE-3475] ShingleFilter should handle positionIncrement of zero, e.g. synonyms

• Type: New Feature
• Status: Patch Available
• Priority: Minor
• Resolution: Unresolved
• Component/s: modules/analysis
• Labels: newdev
• Lucene Fields: New
• Assignee: Alan Woodward
• Reporter: Cameron
• Created: 28/Sep/11 20:41
• Updated: 13/Mar/18 02:22

Description

ShingleFilter is creating shingles for a single term that has been expanded by synonyms when it shouldn't. The position increment is 0.

As an example, I have an Analyzer with a SynonymFilter followed by a ShingleFilter. Assuming car and auto are synonyms, the SynonymFilter produces two tokens and position 1: car, auto. The ShingleFilter is then producing 3 tokens, when there should only be two: car, car auto, auto. This behavior seems incorrect.

Issue Links

• is part of: LUCENE-4170 TestRandomChains fail with Shingle+CommonGrams