[LUCENE-8904] Enhance Nori DictionaryBuilder tool

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 8.x, master (9.0)
• Lucene Fields: New
• Assignee: Namgyu Kim
• Reporter: Namgyu Kim
• Created: 07/Jul/19 14:49
• Updated: 08/Aug/19 16:38
• Resolved: 08/Aug/19 16:38

Description

It is the Nori version of Michael Sokolov's LUCENE-8863.
This patch has two changes.
1) Improve exception handling
2) Enable external dictionary for testing

Overall, it is the same as LUCENE-8863.

But there are some differences between Nori and Kuromoji.
These can be slightly different on the code.
1) CSV field size
Nori : 12
Kuromoji : 13
2) left context ID == right context ID
Nori : can be different
Kuromoji : always same
3) Dictionary Type
Nori : just one type
Kuromoji : IPADIC, UNIDIC

After this job, I'll apply LUCENE-8866 and LUCENE-8871 to Nori.

Issue Links

• links to: GitHub Pull Request #767