[LUCENE-3361] port url+email tokenizer to standardtokenizerinterface (or similar)

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3
• Fix Version/s: 3.4, 4.0-ALPHA
• Component/s: modules/analysis
• Lucene Fields: New
• Assignee: Robert Muir
• Reporter: Robert Muir
• Created: 04/Aug/11 20:02
• Updated: 28/Aug/22 12:54
• Resolved: 08/Aug/11 12:17

Description

We should do this so that we can fix the LUCENE-3358 bug there, and preserve backwards.
We also want this mechanism anyway, for upgrading to new unicode versions in the future.

We can regenerate the new TLD list for 3.4 but, we should ensure the existing one is used for the urlemail33 or whatever,
so that its exactly the same.

Issue Links

• relates to: LUCENE-3358 StandardTokenizer disposes of Hiragana combining mark dakuten instead of attaching it to the character it belongs to