[LUCENE-6837] Add N-best output capability to JapaneseTokenizer

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 5.3
• Fix Version/s: 6.0
• Component/s: modules/analysis
• Lucene Fields: New
• Assignee: Christian Moen
• Reporter: KONNO, Hiroharu
• Created: 13/Oct/15 05:29
• Updated: 28/Aug/22 14:44
• Resolved: 18/Feb/17 00:13

Description

Japanese morphological analyzers often generate mis-segmented tokens. N-best output reduces the impact of mis-segmentation on search result. N-best output is more meaningful than character N-gram, and it increases hit count too.

If you use N-best output, you can get decompounded tokens (ex: "シニアソフトウェアエンジニア" =>

{"シニア", "シニアソフトウェアエンジニア", "ソフトウェア", "エンジニア"}

) and overwrapped tokens (ex: "数学部長谷川" =>

{"数学", "部", "部長", "長谷川", "谷川"}

), depending on the dictionary and N-best parameter settings.