Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
-
New
Description
While building a custom Kuromoji system dictionary, I discovered a few issues.
First, the dictionary encoding has room for 13-bit (left and right) ids, but really only supports 12 bits since this was all that was needed for the IPADIC dictionary that ships with Kuromoji. The good news is we can easily add support by fixing the bit-twiddling math.
Second, the dictionary builder has a number of assertions that help uncover problems in the input (like these overlarge ids), but the assertions aren't enabled by default, so an unsuspecting new user doesn't get any benefit from them, so we should upgrade to "real" exceptions.
Finally, we want to handle the case of empty base forms differently. Kuromoji does stemming by substituting a base form for a word when there is a base form in the dictionary. Missing base forms are expected to be supplied as {}, but if a dictionary provides an empty string base form, we would end up stripping that token completely. Since there is no possible meaning for an empty base form (and the dictionary builder already treats {} and empty strings as equivalent in a number of other cases), I think we should simply ignore empty base forms (rather than replacing words with empty strings when tokenizing!)
Attachments
Issue Links
- links to
