[SOLR-1078] WordDelimiterFilter do wrong word breaking for Thai vowel

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4
• Fix Version/s: 1.4
• Component/s: Schema and Analysis

Environment

Ubuntu 8.10 64bit
Java 1.6.0_10

• Assignee: Unassigned
• Reporter: SIriwat Aumngamsup
• Created: 20/Mar/09 12:48
• Updated: 10/Nov/09 15:51
• Resolved: 08/May/09 20:26

Description

With any configuration of schema.xml

<filter class="solr.WordDelimiterFilterFactory" />

will do wrong word breaking with Thai characters.

---

Example: "ผู้ ใหญ่ บ้าน"

Wrong result: 0 => "ผ", 1 => "ใหญ", 2 => "บ", 3 => "าน"

Expect result: 0 => "ผู้", 1 => "ใหญ่", 2 => "บ้าน"

---

Example2: "ผู้ใหญ่บ้าน" (no space)

Wrong result: 0 => "ผ", 1 => "ใหญ", 2 => "บ", 3 => "าน" (same result)

Expect result: 0 => "ผู้ใหญ่บ้าน"

---

There's a similar problem with Drupal (http://drupal.org/node/335928)