Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-8278

UAX29URLEmailTokenizer is not detecting some tokens as URL type

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • None
    • 7.4, 8.0
    • None
    • None
    • New

    Description

      We are using the UAX29URLEmailTokenizer so we can use the token types in our plugins.

      However, I noticed that the tokenizer is not detecting certain URLs as <URL> but <ALPHANUM> instead.

      Examples that are not working:

      • example.com is <ALPHANUM>
      • example.net is <ALPHANUM>

      But:

      Examples that work:

      • example.ch is <URL>
      • example.co.uk is <URL>
      • example.nl is <URL>

      I have checked this JIRA, and could not find an issue. I have tested this on Lucene (Solr) 6.4.1 and 7.3.

      Could someone confirm my findings and advise what I could do to (help) resolve this issue?

      Attachments

        1. LUCENE-8278.patch
          3.11 MB
          Steven Rowe
        2. unpatched.png
          76 kB
          Steven Rowe
        3. patched.png
          75 kB
          Steven Rowe

        Issue Links

          Activity

            People

              sarowe Steven Rowe
              drjz Junte Zhang
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: