[LUCENE-324] org.apache.lucene.analysis.cn.ChineseTokenizer missing offset decrement

• Type: Bug
• Status: Closed
• Priority: Trivial
• Resolution: Fixed
• Fix Version/s: 1.9
• Component/s: modules/analysis

Environment

Operating System: All
Platform: All

• Bugzilla Id: 32687
• Assignee: Unassigned
• Reporter: Ray Tsang
• Created: 14/Dec/04 18:09
• Updated: 28/Aug/22 11:19
• Resolved: 05/Dec/05 08:09

Description

Apparently, in ChineseTokenizer, offset should be decremented like bufferIndex
when Character is OTHER_LETTER. This directly affects startOffset and endOffset
values.

This is critical to have Highlighter working correctly because Highlighter marks
matching text based on these offset values.