[LUCENE-2206] integrate snowball stopword lists

• Type: New Feature
• Status: Reopened
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0-ALPHA
• Component/s: modules/analysis
• Lucene Fields: New, Patch Available
• Assignee: Robert Muir
• Reporter: Robert Muir
• Created: 13/Jan/10 14:35
• Updated: 27/Aug/24 15:35
• Resolved: 16/Jan/10 14:07

Description

The snowball project creates stopword lists as well as stemmers, example: http://svn.tartarus.org/snowball/trunk/website/algorithms/english/stop.txt?view=markup

This patch includes the following:

• snowball stopword lists for 13 languages in contrib/snowball/resources
• all stoplists are unmodified, only added license header and converted each one from whatever encoding it was in to UTF-8
• added getSnowballWordSet to WordListLoader, this is because the format of these files is very different, for example it supports multiple words per line and embedded comments.

I did not add any changes to SnowballAnalyzer to actually automatically use these lists yet, i would like us to discuss this in a future issue proposing integrating snowball with contrib/analyzers.

Issue Links

• is depended upon by: LUCENE-2055 Fix buggy stemmers and Remove duplicate analysis functionality