[SOLR-1850] KeepWordFilter can be slow at query time if wordlist is large

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4
• Fix Version/s: 3.1, 4.0-ALPHA
• Component/s: Schema and Analysis
• Assignee: Unassigned
• Reporter: John Wang
• Created: 26/Mar/10 21:34
• Updated: 30/Mar/11 15:45
• Resolved: 27/Mar/10 03:44

Description

In the case when "Set<String> words" is large, constructing a KeepWordFilter at query time is very costly because of the construction (copy) of the set, e.g.:

this.words = new CharArraySet(words, ignoreCase);

This call does an addAll on the set, and is done for each query, and is the same work.

Suggestion: overload the constructor and expose the CharArraySet, e.g.:

public KeepWordFilter(TokenStream in, CharArraySet words )

{ super(in); this.words = words; this.termAtt = (TermAttribute)addAttribute(TermAttribute.class); }

This allows the ability to have CharArraySet to be constructed once staticly for the application instead at query time.