[SOLR-11976] TokenizerChain is overwriting, not chaining TokenFilters in normalize()

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 8.0
• Fix Version/s: 7.3
• Component/s: search
• Assignee: David Smiley
• Reporter: Tim Allison
• Created: 12/Feb/18 21:08
• Updated: 08/Jun/19 15:14
• Resolved: 09/Mar/18 03:35

Description

TokenizerChain is overwriting, not chaining tokenfilters in normalize.

This doesn't currently break search because normalize is not being used at the Solr level (AFAICT); rather, TextField has its own analyzeMultiTerm() that duplicates code from the newer normalize.

Code as is:

    TokenStream result = in;
    for (TokenFilterFactory filter : filters) {
      if (filter instanceof MultiTermAwareComponent) {
        filter = (TokenFilterFactory) ((MultiTermAwareComponent) filter).getMultiTermComponent();
        result = filter.create(in);
      }
    }

The fix is simple:

-        result = filter.create(in);
+        result = filter.create(result);

Issue Links

• relates to: SOLR-12034 Replace TokenizerChain in Solr with Lucene's CustomAnalyzer
• links to: GitHub Pull Request #322
• links to: GitHub Pull Request #322