[LUCENE-2901] KeywordMarkerFilter resets keyword attribute state to false for tokens not in protwords.txt

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.1
• Fix Version/s: 3.1, 4.0-ALPHA
• Component/s: modules/analysis
• Lucene Fields: New
• Assignee: Robert Muir
• Reporter: Drew Farris
• Created: 30/Jan/11 17:49
• Updated: 28/Aug/22 12:40
• Resolved: 31/Jan/11 14:46

Description

KeywordMarkerFilter sets true or false for the KeywordAttribute on all tokens. This erases previous state established further up the filter chain, for example in the case where a custom filter wants to prevent a token from being stemmed.

If a token is already marked as a keyword (KeywordAttribute.isKeyword() == true), perhaps the KeywordMarkerFilterFactory should not re-set the state to false.