[LUCENE-3747] Support Unicode 6.1.0

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.5, 4.0-ALPHA
• Fix Version/s: 4.0-BETA, 6.0
• Component/s: modules/analysis
• Lucene Fields: New, Patch Available
• Assignee: Steven Rowe
• Reporter: Steven Rowe
• Created: 02/Feb/12 17:53
• Updated: 28/Aug/22 13:07
• Resolved: 20/Sep/12 01:28

Description

Now that Unicode 6.1.0 has been released, Lucene/Solr should support it.

JFlex trunk now supports Unicode 6.1.0.

Tasks include:

• Upgrade ICU4J to v49 (after it's released, on 2012-03-21, according to http://icu-project.org).
• Use icu module tools to regenerate the supplementary character additions to JFlex grammars.
• Version the JFlex grammars: copy the current implementations to *Impl3<X>; cause the versioning tokenizer wrappers to instantiate this version when the Version c-tor param is in the range 3.1 to the version in which these changes are released (excluding the range endpoints); then change the specified Unicode version in the non-versioned JFlex grammars from 6.0 to 6.1.
• Regenerate JFlex scanners, including StandardTokenizerImpl, UAX29URLEmailTokenizerImpl, and HTMLStripCharFilter.
• Using generateJavaUnicodeWordBreakTest.pl, generate and then run WordBreakTestUnicode_6_1_0.java under modules/analysis/common/src/test/org/apache/lucene/analysis/core/