[SOLR-2845] Adding extra highlighting term to a synonym

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.4
• Fix Version/s: 3.4
• Component/s: highlighter

Environment

Solr release: 3.4.0
JVM:
java version "1.6.0_16"
Java(TM) SE Runtime Environment (build 1.6.0_16-b01)
Java HotSpot(TM) 64-Bit Server VM (build 14.2-b01, mixed mode)
OS: 2.6.18-274.el5 #1 SMP Fri Jul 8 17:36:59 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

• Assignee: Unassigned
• Reporter: Ajay Kanduru
• Created: 18/Oct/11 15:09
• Updated: 21/Nov/11 10:19

Description

I notice a strange highlighting behaviour while highlighting a synonym term. It is in 3.4.0 release. This is working fine in 1.4.1. Using solr example core, here are the steps to reproduce the problem.

1) In schema.xml, change text_general fieldtype definition to use synonym filter at index time and remove the filter from query analysis.

<fieldType name="text_general" class="solr.TextField" positionIncrementGap="100">
  <analyzer type="index">
    <tokenizer class="solr.StandardTokenizerFactory"/>
    <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="true" />

    <filter class="solr.SynonymFilterFactory" synonyms="index_synonyms.txt" ignoreCase="true" expand="true"/>
    <filter class="solr.LowerCaseFilterFactory"/>
  </analyzer>
  <analyzer type="query">
    <tokenizer class="solr.StandardTokenizerFactory"/>
    <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="true" />
    <!-- <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/> -->

    <filter class="solr.LowerCaseFilterFactory"/>

  </analyzer>
</fieldType>

2) Define a new field 'test_field1'.

  <field name="test_field1" type="text_general" indexed="true" stored="true" multiValued="true"/>

3) Copy this to 'text' field.

  <copyField source="test_field1" dest="text"/>

4) In exampledocs/ipod_video.xml, add a new field to the doc.

  <field name="test_field1">Heart Failure</field>

5) In solr/conf/index_synonyms.txt:, add the following line (all in one line).

heart failure, failure\, heart, cardiac failure, cardiac insufficiency, failure heart, failure\, cardiac, heart failure (nos), insufficiency cardiac, insufficiency\, cardiac, hf - heart failure

6) Reindex exampledocs/*xml files and run the following URL.

http://localhost:8983/solr/select?q=heart&indent=on&hl=on&hl.fl=*

This is what I get from highlighting tag.

  <lst name="highlighting">
    <lst name="MA147LL/A">
      <arr name="test_field1">
        <str>&lt;em&gt;Heart&lt;/em&gt;&lt;em&gt;Heart Failure&lt;/em&gt;</str>
      </arr>
    </lst>
  </lst>

The actual value of the field is Heart Failure. It is changed to HeartHeart Failure.

Apparently the synonym entries has something to do with the problem. The above synonym terms are the minimum extraction from a larger line to reproduce the problem. Notice that there is a hyphen in the last term. If I remove the hyphen, it works, even with larger line of entries. Keeping the hyphen, and removing insufficiency\, cardiac, also works. So the length of the line and hyphen both seem at play here.

Using large and complicated synonyms is very important to our application. 3.4 release has announced some major improvements to memory foot print and performance for synonym filter. For this reason we are eager to move to 3.4.0, but this problem is a show stopper for us. I will appreciate any suggestions for a work around or a quick fix to the problem.

Regards,
-Ajay