[LUCENE-2207] CJKTokenizer generates tokens with incorrect offsets

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.9.1, 3.0
• Fix Version/s: 2.9.2, 3.0.1, 4.0-ALPHA
• Component/s: modules/analysis
• Lucene Fields: New, Patch Available
• Assignee: Robert Muir
• Reporter: Koji Sekiguchi
• Created: 13/Jan/10 16:38
• Updated: 16/May/11 18:15
• Resolved: 17/Jan/10 21:45

Description

If I index a Japanese multi-valued document with CJKTokenizer and highlight a term with FastVectorHighlighter, the output snippets have incorrect highlighted string. I'll attach a program that reproduces the problem soon.

Issue Links

• is part of: LUCENE-2219 improve BaseTokenStreamTestCase to test end()