[SOLR-4945] Japanese Autocomplete and Highlighter broken

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 1.4.1
• Component/s: highlighter
• Assignee: Unassigned
• Reporter: Shruthi Khatawkar
• Created: 21/Jun/13 06:34
• Updated: 02/Oct/16 01:28
• Resolved: 02/Oct/16 01:28

Description

Autocomplete is implemented with Highlighter functionality. This works fine for most of the languages but breaks for Japanese.

multivalued,termVector,termPositions and termOffset are set to true.

Here is an example:

Query: product classic.

Result:

Actual :

この商品の互換性の機種にproduct 1 やclassic Touch2　が記載が有りません。 USB接続ケーブルをproduct 1 やclassic Touch2に付属の物を使えば利用出来ると思いますが 間違っていますか？

With Highlighter (<em> </em> tags being used):

この商品の互換性の機種<em>にproduct</em> 1 <em>やclassic</em> Touch2　が記載が有りません。 USB接続ケーブルをproduct 1 やclassic Touch2に付属の物を使えば利用出来ると思いますが 間違っていますか？

Though query terms "product classic" is repeated twice, highlighting is happening only on the first instance. As shown above.

Solr returns only first instance offset and second instance is ignored.

Also it's observed, highlighter repeats first letter of the token if there is numeric.
For eg.Query : product and We have product1, highlighter returns as p<em>product</em>1.