[LUCENE-8869] Build kuromoji system dictionary as a separated jar and load it from JapaneseTokenizer at runtime

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: modules/analysis
• Lucene Fields: New
• Assignee: Unassigned
• Reporter: Tomoko Uchida
• Created: 19/Jun/19 16:13
• Updated: 28/Aug/22 15:47

Description

This is a sub-task for LUCENE-8816.
In this issue, I will try to make small but self-contained changes to kuromoji system dictionary.

• Make it possible to build a jar that contains (maybe) only dictionary data resource generated by the build-dict task.
  • Maybe a new ant target will be added.
• Make it possible to load external dictionary when initializing JapaneseTokenizer.
  • Some work are already done on LUCENE-8863
• Decouple current system dictionary data (mecab ipadic) from kuromoji itself and use it as default (Possibly it can be done with another issue).

Also, some refactoring of the directory/source tree structure may be needed.

Issue Links

• relates to: LUCENE-8816 Decouple Kuromoji's morphological analyser and its dictionary