[LUCENE-2400] ShingleFilter: don't output all-filler shingles/unigrams; also, convert from TermAttribute to CharTermAttribute

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.0.1
• Fix Version/s: 3.1, 4.0-ALPHA
• Component/s: modules/analysis
• Lucene Fields: New, Patch Available
• Assignee: Uwe Schindler
• Reporter: Steven Rowe
• Created: 18/Apr/10 15:29
• Updated: 28/Aug/22 12:24
• Resolved: 03/May/10 13:30

Description

When the input token stream to ShingleFilter has position increments greater than one, filler tokens are inserted for each position for which there is no token in the input token stream. As a result, unigrams (if configured) and shingles can be filler-only. Filler-only output tokens make no sense - these should be removed.

Also, because TermAttribute has been deprecated in favor of CharTermAttribute, the patch will also convert TermAttribute usages to CharTermAttribute in ShingleFilter.