[LUCENE-3979] NGramTokenizer strips whitespace, with no option to keep leading and trailing whitespace

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 2.9.2, 3.0
• Component/s: modules/analysis
• Labels: tokenizer whitespace

Environment

n/a

• Lucene Fields: New
• Assignee: Unassigned
• Reporter: David Mason
• Created: 13/Apr/12 03:43
• Updated: 28/Aug/22 13:14

Description

org.apache.lucene.analysis.ngram.NGramTokenizer removes whitespace, making a search for literal strings like " test" and "test " equivalent to "test". Searching with relevant whitespace is sometimes desired, particularly where ngrams are used.

This could be fixed by either removing .trim() from the line shown below, or by providing a flag to specifically set trimming behaviour (keeping trim=true as the default so that existing code using this analyzer is not broken).

111: inStr = new String(chars).trim(); // remove any trailing empty strings