[LUCENE-2699] Update StandardTokenizer and UAX29Tokenizer to Unicode 6.0.0

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.1, 4.0-ALPHA
• Fix Version/s: 3.1, 4.0-ALPHA
• Component/s: modules/analysis
• Lucene Fields: New, Patch Available
• Assignee: Steven Rowe
• Reporter: Steven Rowe
• Created: 13/Oct/10 04:30
• Updated: 28/Aug/22 12:34
• Resolved: 15/Oct/10 06:34

Description

Newly released Unicode 6.0.0 contains some character property changes from the previous release (5.2.0) that affect word segmentation (UAX#29), and JFlex 1.5.0-SNAPSHOT now supports Unicode 6.0.0, so Lucene's UAX#29-based tokenizers should be updated accordingly.

Note that the UAX#29 word break rules themselves did not change between Unicode versions 5.2.0 and 6.0.0.