[SOLR-3235] Whitespace issue in synonym list

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 3.4, 3.5
• Fix Version/s: 3.4
• Component/s: Schema and Analysis
• Labels: synonyms

Environment

Windows 7
Firefox 10.0.2
Solr example (start.jar)

• Assignee: Unassigned
• Reporter: Johannes Brucher
• Created: 12/Mar/12 12:43
• Updated: 10/May/13 10:38
• Resolved: 12/Mar/12 13:13

Description

If you use the following schema.xml entrie:

<fieldType name="contenttype" class="solr.TextField" multiValued="true" omitNorms="true">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="false"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
</analyzer>
</fieldType>

With a synonym list having such entrie:

text/html;\ charset=ISO-8859-1 => html

Solr 3.4 and 3.5 can't handle the whitespace between "html;" and "charset" and no synonym substitution is processed. The same config works find in Solr 3.3.
No exception or error is thrown.

This is my first jira ticket, so if I mist something let me know...

Regrads

Johannes

Edit: Ok found the solution for that problem. Provide the following:

<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="false"
tokenizerFactory="solr.KeywordTokenizerFactory" />

As tokenizerFactory you should use "solr.KeywordTokenizerFactory" instead of "solr.WhitespaceTokenizerFactory".

See the javadocs for more details:
https://builds.apache.org/job/Solr-trunk/javadoc/org/apache/solr/analysis/SynonymFilterFactory.html