[LUCENE-1689] supplementary character handling

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Component/s: modules/analysis
• Lucene Fields: New
• Assignee: Unassigned
• Reporter: Robert Muir
• Created: 12/Jun/09 18:35
• Updated: 28/Aug/22 12:02
• Resolved: 12/Jan/13 23:06

Description

for Java 5. Java 5 is based on unicode 4, which means variable-width encoding.

supplementary character support should be fixed for code that works with char/char[]

For example:
StandardAnalyzer, SimpleAnalyzer, StopAnalyzer, etc should at least be changed so they don't actually remove suppl characters, or modified to look for surrogates and behave correctly.
LowercaseFilter should be modified to lowercase suppl. characters correctly.
CharTokenizer should either be deprecated or changed so that isTokenChar() and normalize() use int.

in all of these cases code should remain optimized for the BMP case, and suppl characters should be the exception, but still work.

Issue Links

• incorporates: LUCENE-2847 Support all of unicode in StandardTokenizer
• incorporates: LUCENE-2068 fix reverseStringFilter for unicode 4.0
• incorporates: LUCENE-2183 Supplementary Character Handling in CharTokenizer
• incorporates: LUCENE-2069 fix LowerCaseFilter for unicode 4.0
• incorporates: LUCENE-2070 document LengthFilter wrt Unicode 4.0
• is related to: LUCENE-2094 Prepare CharArraySet for Unicode 4.0