Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-2198

support protected words in Stemming TokenFilters

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 3.0
    • 4.0-ALPHA
    • modules/analysis
    • None
    • New, Patch Available

    Description

      This is from LUCENE-1515

      I propose that all stemming TokenFilters have an 'exclusion set' that bypasses any stemming for words in this set.
      Some stemming tokenfilters have this, some do not.

      This would be one way for Karl to implement his new swedish stemmer (as a text file of ignore words).
      Additionally, it would remove duplication between lucene and solr, as they reimplement snowballfilter since it does not have this functionality.
      Finally, I think this is a pretty common use case, where people want to ignore things like proper nouns in the stemming.

      As an alternative design I considered a case where we generalized this to CharArrayMap (and ignoring words would mean mapping them to themselves), which would also provide a mechanism to override the stemming algorithm. But I think this is too expert, could be its own filter, and the only example of this i can find is in the Dutch stemmer.

      So I think we should just provide ignore with CharArraySet, but if you feel otherwise please comment.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            uschindler Uwe Schindler
            rcmuir Robert Muir
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment