[LUCENE-8325] smartcn analyzer can't handle SURROGATE char

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 7.4, 8.0
• Labels: newbie patch
• Lucene Fields: New
• Assignee: Unassigned
• Reporter: chengpohi
• Created: 22/May/18 02:16
• Updated: 28/Aug/22 15:30
• Resolved: 23/May/18 14:20

Description

This issue is from https://github.com/elastic/elasticsearch/issues/30739

smartcn analyzer can't handle SURROGATE char, Example:

 

 

Analyzer ca = new SmartChineseAnalyzer(); 
String sentence = "\uD862\uDE0F"; // 𨨏 a surrogate char 
TokenStream tokenStream = ca.tokenStream("", sentence); 
CharTermAttribute charTermAttribute = tokenStream.addAttribute(CharTermAttribute.class); 
tokenStream.reset(); 
while (tokenStream.incrementToken()) { 
    String term = charTermAttribute.toString(); 
    System.out.println(term); 
} 

 

In the above code snippet will output: 

 

? 
? 

 

 and I have created a PATCH to try to fix this, please help review(since smartcn only support GBK char, so it's only just handle it as a single char).