[LUCENE-1556] some valid email address characters not correctly recognized

• Type: Bug
• Status: Closed
• Priority: Trivial
• Resolution: Fixed
• Affects Version/s: 2.4.1
• Fix Version/s: 3.1, 4.0-ALPHA
• Component/s: modules/analysis
• Lucene Fields: New
• Assignee: Unassigned
• Reporter: Paul Nilsson
• Created: 10/Mar/09 15:15
• Updated: 28/Aug/22 11:58
• Resolved: 29/Sep/10 05:50

Description

the EMAIL expression in StandardTokenizerImpl.jflex misses some unusual but valid characters in the left-hand-side of the email address. This causes an address to be broken into several tokens, for example:

[e-mail] gets broken into "somename" and "[e-mail]"
husband&[e-mail] gets broken into "husband" and "[e-mail]"

These seem to be occurring more often. The first seems to be because of an anti-spam trick you can use with google (see: http://labnol.blogspot.com/2007/08/gmail-plus-smart-trick-to-find-block.html). I see the second in several domains but a disproportionate amount are from talktalk.net, so I expect it's a signup suggestion from the service.

Perhaps a fix would be to change line 102 of StandardTokenizerImpl.jflex from:
EMAIL =

{ALPHANUM} (("."|"-"|"_") {ALPHANUM}

)* "@"

{ALPHANUM} (("."|"-") {ALPHANUM}

)+

to

EMAIL =

{ALPHANUM} (("."|"-"|"_"|"+"|"&") {ALPHANUM}

)* "@"

{ALPHANUM} (("."|"-") {ALPHANUM}

)+

I'm aware that the StandardTokenizer is meant to be more of a basic implementation rather than an implementation the full standard, but it is quite useful in places and hopefully this would improve it slightly.

Issue Links

• is part of: LUCENE-2167 Implement StandardTokenizer with the UAX#29 Standard