[LUCENE-2167] Implement StandardTokenizer with the UAX#29 Standard

• Type: New Feature
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.1, 4.0-ALPHA
• Fix Version/s: 3.1, 4.0-ALPHA
• Component/s: modules/analysis
• Lucene Fields: Patch Available
• Assignee: Steven Rowe
• Reporter: Shyamal Prasad
• Created: 16/Dec/09 18:48
• Updated: 28/Aug/22 12:17
• Resolved: 15/Nov/10 18:27

Description

It would be really nice for StandardTokenizer to adhere straight to the standard as much as we can with jflex. Then its name would actually make sense.

Such a transition would involve renaming the old StandardTokenizer to EuropeanTokenizer, as its javadoc claims:

This should be a good tokenizer for most European-language documents

The new StandardTokenizer could then say

This should be a good tokenizer for most languages.

All the english/euro-centric stuff like the acronym/company/apostrophe stuff can stay with that EuropeanTokenizer, and it could be used by the european analyzers.

Issue Links

• incorporates: LUCENE-1545 Standard analyzer does not correctly tokenize combining character U+0364 COMBINING LATIN SMALL LETTRE E
• incorporates: LUCENE-1702 Thai token type() bug
• incorporates: LUCENE-1556 some valid email address characters not correctly recognized
• is related to: LUCENE-2763 Swap URL+Email recognizing StandardTokenizer and UAX29Tokenizer
• relates to: LUCENE-2244 Improve StandardTokenizer's understanding of non ASCII punctuation and quotes