[LUCENE-8784] Nori(Korean) tokenizer removes the decimal point.

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 9.0, 8.2
• Lucene Fields: New
• Assignee: Unassigned
• Reporter: Munkyu Im
• Created: 30/Apr/19 04:06
• Updated: 28/Aug/22 15:44
• Resolved: 27/May/19 13:31

Description

This is the same issue that I mentioned to https://github.com/elastic/elasticsearch/issues/41401#event-2293189367

unlike standard analyzer, nori analyzer removes the decimal point.

nori tokenizer removes "." character by default.
In this case, it is difficult to index the keywords including the decimal point.

It would be nice if there had the option whether add a decimal point or not.

Like Japanese tokenizer does,  Nori need an option to preserve decimal point.

 

Issue Links

• relates to: LUCENE-8812 add KoreanNumberFilter to Nori(Korean) Analyzer