[LUCENE-7960] NGram filters -- preserve the original token when it is outside the min/max size range

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 7.4, 8.0
• Component/s: modules/analysis
• Lucene Fields: New
• Assignee: Robert Muir
• Reporter: Shawn Heisey
• Created: 06/Sep/17 18:25
• Updated: 28/Aug/22 15:19
• Resolved: 05/Jun/18 02:42

Description

When ngram or edgengram filters are used, any terms that are shorter than the minGramSize are completely removed from the token stream.

This is probably 100% what was intended, but I've seen it cause a lot of problems for users. I am not suggesting that the default behavior be changed. That would be far too disruptive to the existing user base.

I do think there should be a new boolean option, with a name like keepShortTerms, that defaults to false, to allow the short terms to be preserved.

Issue Links

• is duplicated by: SOLR-5152 EdgeNGramFilterFactory deletes token
• links to: GitHub Pull Request #362