[LUCENE-973] Token of "" returns in CJKTokenizer + new TestCJKTokenizer

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.3
• Fix Version/s: 2.9
• Component/s: modules/analysis
• Lucene Fields: New, Patch Available
• Assignee: Michael McCandless
• Reporter: Toru Matsuzawa
• Created: 07/Aug/07 13:31
• Updated: 28/Aug/22 11:40
• Resolved: 16/Jun/09 16:38

Description

The "" string returns as Token in the boundary of two byte character and one byte character.

There is no problem in CJKAnalyzer.
When CJKTokenizer is used with the unit, it becomes a problem. (Use it with
Solr etc.)