Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-5211

StopFilterFactory docs do not advertise/explain hte "format" option

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 4.2
    • 4.6, 6.0
    • None
    • None
    • New

    Description

      StopFilterFactory supports a "format" option for controlling wether "getWordSet" or "getSnowballWordSet" is used to parse the file, but this option is not advertised and people can be confused by looking at the example stopword files include in the releases (some of which are in the snoball format w/ "|" comments) and try to use them w/o explicitly specifying format="snowball" and silently get useless stopwords (that include the "| comments" as literal portions of hte stopwrds.

      we need to better document the use of "format" and consider updating all of the example stopword files we ship that are in the snowball format with a note about the need to use format="snowball" with those files.

      Initial Bug Report

      The StopFilterFactory builds a CharArraySet directly from the raw lines of the supplied words file. This causes a problem when using the stop word files supplied with the Solr/Lucene distribution. In particular, the comments in those files get added to the CharArraySet. A line like this...

      ceci | this

      Should result in the string "ceci" being added to the CharArraySet, but "ceci | this" is what actually gets added.

      Workaround: Remove all comments from stop word files you are using.

      Suggested fix: The StopFilterFactory should strip any comments, then strip trailing whitespace. The stop word files supplied with the distribution should be edited to conform to the supported comment format.

      Attachments

        1. LUCENE-5211.stopfilecomments.patch
          25 kB
          Chris M. Hostetter
        2. LUCENE-5211.code.patch
          6 kB
          Chris M. Hostetter

        Activity

          People

            hossman Chris M. Hostetter
            haydenmuhl Hayden Muhl
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment