[LUCENE-9853] Use CJKWidthCharFilter as the default character normalizer for JapaneseAnalyzer instead of CJKWidthFilter

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 9.0
• Fix Version/s: 9.0
• Component/s: modules/analysis
• Lucene Fields: New
• Assignee: Tomoko Uchida
• Reporter: Tomoko Uchida
• Created: 21/Mar/21 08:46
• Updated: 28/Aug/22 16:18
• Resolved: 29/Oct/21 02:57

Description

Follow-up issue of LUCENE-9413.

We now have CJKWidthCharFilter in analyzers-common. I believe in many situations it is recommended applying half-width/full-width character normalization before tokenization for consistency in analysis.

The change slightly affects on the analyzer's outputs. We can provide a parameter to switch back to CJKWidthFilter for backward compatibility.

Issue Links

• is related to: LUCENE-9413 Add a char filter corresponding to CJKWidthFilter
• links to: GitHub Pull Request #26