[LUCENE-10497] Unify "Token" interface in Kuromoji and Nori

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 10.0 (main)
• Lucene Fields: New
• Assignee: Tomoko Uchida
• Reporter: Tomoko Uchida
• Created: 04/Apr/22 10:54
• Updated: 28/Aug/22 16:38
• Resolved: 18/Jul/22 08:10

Description

"Token" classes are the output of the viterbi algorithm in kuromoji and nori, and its implementation is slightly different between them (Nori's Token class looks more expressive or extendable).

To proceed LUCENE-10493, I think we need the unified "Token" class or at least a common interface for it.

Issue Links

• links to: GitHub Pull Request #783