[LUCENE-400] NGramFilter -- construct n-grams from a TokenStream

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 2.4
• Component/s: modules/analysis

Environment

Operating System: All
Platform: All

• Lucene Fields: Patch Available
• Bugzilla Id: 35456
• Assignee: Grant Ingersoll
• Reporter: Sebastian Kirsch
• Created: 22/Jun/05 06:08
• Updated: 28/Aug/22 11:22
• Resolved: 29/Mar/08 21:09

Description

This filter constructs n-grams (token combinations up to a fixed size, sometimes
called "shingles") from a token stream.

The filter sets start offsets, end offsets and position increments, so
highlighting and phrase queries should work.

Position increments > 1 in the input stream are replaced by filler tokens
(tokens with termText "_" and endOffset - startOffset = 0) in the output
n-grams. (Position increments > 1 in the input stream are usually caused by
removing some tokens, eg. stopwords, from a stream.)

The filter uses CircularFifoBuffer and UnboundedFifoBuffer from Apache
Commons-Collections.

Filter, test case and an analyzer are attached.