[LUCENE-8817] Combine Nori and Kuromoji DictionaryBuilder

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Lucene Fields: New
• Assignee: Unassigned
• Reporter: Namgyu Kim
• Created: 01/Jun/19 16:39
• Updated: 28/Aug/22 15:45

Description

This issue is related to LUCENE-8816.

Currently Nori and Kuromoji Analyzer use the same dictionary structure. (MeCab)
If we make combine DictionaryBuilder, we can reduce the code size.
But this task may have a dependency on the language.
(like HEADER string in BinaryDictionary and CharacterDefinition, methods in BinaryDictionaryWriter, ...)
On the other hand, there are many overlapped classes.

The purpose of this patch is to provide users of Nori and Kuromoji with the same system dictionary generator.

It may take some time because there is a little workload.
The work will be based on the latest master, and if the LUCENE-8816 is finished first, I will pull the latest code and proceed.

Issue Links

• is related to: LUCENE-8816 Decouple Kuromoji's morphological analyser and its dictionary