[LUCENE-2763] Swap URL+Email recognizing StandardTokenizer and UAX29Tokenizer

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1, 4.0-ALPHA
• Fix Version/s: 3.1, 4.0-ALPHA
• Component/s: modules/analysis
• Assignee: Steven Rowe
• Reporter: Steven Rowe
• Created: 15/Nov/10 18:24
• Updated: 28/Aug/22 12:36
• Resolved: 07/Dec/10 19:52

Description

Currently, in addition to implementing the UAX#29 word boundary rules, StandardTokenizer recognizes email adresses and URLs, but doesn't provide a way to turn this behavior off and/or provide overlapping tokens with the components (username from email address, hostname from URL, etc.).

UAX29Tokenizer should become StandardTokenizer, and current StandardTokenizer should be renamed to something like UAX29TokenizerPlusPlus (or something like that).

For rationale, see the discussion at the reopened LUCENE-2167.

Issue Links

• relates to: LUCENE-2167 Implement StandardTokenizer with the UAX#29 Standard
• relates to: SOLR-2211 Create Solr FilterFactory for Lucene StandardTokenizer with UAX#29 support