Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-8278

UAX29URLEmailTokenizer is not detecting some tokens as URL type

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 7.4, 8.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      We are using the UAX29URLEmailTokenizer so we can use the token types in our plugins.

      However, I noticed that the tokenizer is not detecting certain URLs as <URL> but <ALPHANUM> instead.

      Examples that are not working:

      • example.com is <ALPHANUM>
      • example.net is <ALPHANUM>

      But:

      Examples that work:

      • example.ch is <URL>
      • example.co.uk is <URL>
      • example.nl is <URL>

      I have checked this JIRA, and could not find an issue. I have tested this on Lucene (Solr) 6.4.1 and 7.3.

      Could someone confirm my findings and advise what I could do to (help) resolve this issue?

        Attachments

        1. unpatched.png
          76 kB
          Steven Rowe
        2. patched.png
          75 kB
          Steven Rowe
        3. LUCENE-8278.patch
          3.11 MB
          Steven Rowe

          Issue Links

            Activity

              People

              • Assignee:
                sarowe Steven Rowe
                Reporter:
                drjz Junte Zhang
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: