Details
-
Improvement
-
Status: Open
-
Trivial
-
Resolution: Unresolved
-
None
-
None
-
New, Patch Available
Description
Korean(nori) and Japanese(kuromoji) analyzers behave the same way by using a dictionary-based and finite-state-based approach to identify words (aka tokens).
When analyzing Korean/Japanese inputs, it needs to perform a lookup in the dictionary on every character in order to build the lattice of all possible segmentations. In order to achieve this efficiently, we encode the full vocabulary in FST (finite state transducer). So we can analyze text using the Viterbi algorithm to find the most likely segmentation (called the Viterbi path) of any input written in Korean or Japanese.
org.apache.lucene.analysis.ko.GraphvizFormatter org.apache.lucene.analysis.ja.GraphvizFormatter
Those two already have Graphviz to visualize the Viterbi lattice built from input texts. However, according to my experience, part of speech is significant to diagnose why the outputs look like since this works with the dictionary-based approach.
Adding tokens' part of speech will help users to understand the analyzers. I and some users are using those classes during their Lucene-related projects although it's a very trivial part. Opening a PR after issue review.