[LUCENE-2911] synchronize grammar/token types across StandardTokenizer, UAX29EmailURLTokenizer, ICUTokenizer, add CJK types.

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.1, 4.0-ALPHA
• Component/s: modules/analysis
• Lucene Fields: New, Patch Available
• Assignee: Robert Muir
• Reporter: Robert Muir
• Created: 08/Feb/11 12:58
• Updated: 28/Aug/22 12:40
• Resolved: 09/Feb/11 17:48

Description

I'd like to do LUCENE-2906 (better cjk support for these tokenizers) for a future target such as 3.2

But, in 3.1 I would like to do a little cleanup first, and synchronize all these token types, etc.