[LUCENE-8278] UAX29URLEmailTokenizer is not detecting some tokens as URL type

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 7.4, 8.0
• Lucene Fields: New
• Assignee: Steven Rowe
• Reporter: Junte Zhang
• Created: 26/Apr/18 09:26
• Updated: 28/Aug/22 15:29
• Resolved: 12/Jun/18 13:36

Description

We are using the UAX29URLEmailTokenizer so we can use the token types in our plugins.

However, I noticed that the tokenizer is not detecting certain URLs as <URL> but <ALPHANUM> instead.

Examples that are not working:

• example.com is <ALPHANUM>
• example.net is <ALPHANUM>

But:

• https://example.com is <URL>
• as is https://example.net

Examples that work:

• example.ch is <URL>
• example.co.uk is <URL>
• example.nl is <URL>

I have checked this JIRA, and could not find an issue. I have tested this on Lucene (Solr) 6.4.1 and 7.3.

Could someone confirm my findings and advise what I could do to (help) resolve this issue?

Issue Links

• is broken by: LUCENE-5391 UAX29URLEmailTokenizer should not tokenize no-scheme domain-only URLs that are followed by an alphanumeric character