Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-2211

Create Solr FilterFactory for Lucene StandardTokenizer with UAX#29 support

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.1
    • Fix Version/s: 3.1, 4.0-ALPHA
    • Component/s: None
    • Labels:
      None

      Description

      The Lucene 3.x StandardTokenizer with UAX#29 support provides benefits for non-English tokenizing. Presently it can be invoked by using the StandardTokenizerFactory and setting the Version to 3.1. However, it would be useful to be able to use the improved unicode processing without necessarily including the ip address and email address processing of StandardAnalyzer. A FilterFactory that allowed the use of the StandardTokenizer with UAX#29 support on its own would be useful.

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              rcmuir Robert Muir
              Reporter:
              tburtonwest Tom Burton-West

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment