[LUCENE-1403] StandardTokenizer - Improper Hostname Recognition

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 2.3.1, 2.3.2
• Component/s: modules/analysis

Environment

Java 5

• Lucene Fields: New
• Assignee: Unassigned
• Reporter: Cullin Wible
• Created: 23/Sep/08 17:43
• Updated: 28/Aug/22 11:53
• Resolved: 26/Jan/11 12:07

Description

As of 2.3.1 the documentation for the StandardTokenizer states that it "Recognizes email addresses and internet hostnames as one token."

However hostnames such as "my-host.com" are recognized as two tokens "my" and "host.com".

Any host with a dash in the name is not recognized properly.

Issue Links

• is related to: LUCENE-1373 Most of the contributed Analyzers suffer from invalid recognition of acronyms.