[LUCENE-2906] Filter to process output of ICUTokenizer and create overlapping bigrams for CJK

• Type: New Feature
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 3.6, 4.0-ALPHA
• Component/s: modules/analysis
• Lucene Fields: New
• Assignee: Robert Muir
• Reporter: Tom Burton-West
• Created: 04/Feb/11 20:30
• Updated: 28/Aug/22 12:40
• Resolved: 29/Dec/11 05:24

Description

The ICUTokenizer produces unigrams for CJK. We would like to use the ICUTokenizer but have overlapping bigrams created for CJK as in the CJK Analyzer. This filter would take the output of the ICUtokenizer, read the ScriptAttribute and for selected scripts (Han, Kana), would produce overlapping bigrams.

Sub-Tasks

• synchronize grammar/token types across StandardTokenizer, UAX29EmailURLTokenizer, ICUTokenizer, add CJK types.: Closed