[LUCENE-461] StandardTokenizer splitting all of Korean words into separate characters

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 1.9
• Component/s: modules/analysis

Environment

Analyzing Korean text with Apache Lucene, esp. with StandardAnalyzer.

• Assignee: Unassigned
• Reporter: Cheolgoo Kang
• Created: 08/Nov/05 15:55
• Updated: 28/Aug/22 11:24
• Resolved: 12/Nov/05 17:36

Description

StandardTokenizer splits all those Korean words inth separate character tokens. For example, "?????" is one Korean word that means "Hello", but StandardAnalyzer separates it into five tokens of "?", "?", "?", "?", "?".