[LUCENENET-573] Make IcuBreakIterator more like the JDK's BreakIterator.getInstance()

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: Lucene.Net 4.8.0
• Component/s: Lucene.Net.ICU
• Assignee: Unassigned
• Reporter: Shad Storhaug
• Created: 06/Apr/17 07:14
• Updated: 13/Aug/19 05:33
• Resolved: 13/Aug/19 05:33

Description

The IcuBreakIterator is a wrapper around the icu-dotnet library. It implements the JDK BreakIterator business logic that was previously missing there, but has since been added in the form of a RuleBasedBreakIterator. IcuBreakIterator is utilized by Lucene.Net.Analysis.Common.Th.ThaiAnalyzer, Lucene.Net.Highlighter.PostingsHighlight, and Lucene.Net.Highlighter.VectorHighlight. While all of the tests are passing for these components, it is primarily because of hacks that were added as workarounds. In reality, the functionality of IcuBreakIterator has many rule-based differences that make its breaking text behavior act quite differently than the JDK.

We need to investigate whether the RuleBasedBreakIterator in icu-dotnet can be utilized as is, or if it can be improved to more closely emulate the BreakIterator functionality in the JDK.