[LUCENE-571] StandardTokenizer parses decimal number as <HOST>

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 1.9
• Component/s: modules/analysis
• Assignee: Unassigned
• Reporter: Tom Emerson
• Created: 16/May/06 02:44
• Updated: 28/Aug/22 11:27
• Resolved: 01/Jan/08 11:45

Description

The standard tokenizer in 1.9.1 returns a decimal number such as "3.14" as a <HOST>, though a number like "3,141.59" is returned as a <NUM>. I believe, though I haven't tried it yet, that moving the rule for <HOST> after <NUM>, instead of before it, will obviate this. Or updating <HOST> to require a TLD as the last component, which would require you to split the interpretation of IP addresses from name-based addresses.

Issue Links

• is related to: LUCENE-1100 StandardTokenizer incorrectly types certain values