[LUCENE-1229] NGramTokenFilter optimization in query phase

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Component/s: modules/analysis
• Lucene Fields: New, Patch Available
• Assignee: Otis Gospodnetic
• Reporter: Hiroaki Kawai
• Created: 14/Mar/08 02:30
• Updated: 28/Aug/22 11:48
• Resolved: 10/Mar/13 13:30

Description

I found that NGramTokenFilter-ed token stream could be optimized in query.

A standard 1,2 NGramTokenFilter will generate a token stream from "abcde" as follows:
a ab b bc c cd d de e

When we index "abcde", we'll use all of the tokens.

But when we query, we only need:
ab cd de