[LUCENE-3767] Explore streaming Viterbi search in Kuromoji

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.6, 4.0-ALPHA
• Component/s: modules/analysis
• Lucene Fields: New
• Assignee: Christian Moen
• Reporter: Michael McCandless
• Created: 09/Feb/12 23:39
• Updated: 28/Aug/22 13:07
• Resolved: 10/Mar/12 15:56

Description

I've been playing with the idea of changing the Kuromoji viterbi
search to be 2 passes (intersect, backtrace) instead of 4 passes
(break into sentences, intersect, score, backtrace)... this is very
much a work in progress, so I'm just getting my current state up.
It's got tons of nocommits, doesn't properly handle the user dict nor
extended modes yet, etc.

One thing I'm playing with is to add a double backtrace for the long
compound tokens, ie, instead of penalizing these tokens so that
shorter tokens are picked, leave the scores unchanged but on backtrace
take that penalty and use it as a threshold for a 2nd best
segmentation...

Issue Links

• is depended upon by: LUCENE-3843 implement PositionLengthAttribute for all tokenstreams where its appropriate