[SOLR-1869] RemoveDuplicatesTokenFilter doest have expected behaviour

• Type: New Feature
• Status: Closed
• Priority: Minor
• Resolution: Not A Problem
• Component/s: Schema and Analysis
• Labels: RemoveDuplicatesTokenFilter
• Assignee: Unassigned
• Reporter: Joe Calderon
• Created: 07/Apr/10 16:59
• Updated: 22/Apr/10 21:12
• Resolved: 22/Apr/10 21:12

Description

the RemoveDuplicatesTokenFilter seems broken as it initializes its map and attributes at the class level and not within its constructor
in addition i would think the expected behaviour would be to remove identical terms with the same offset positions, instead it looks like it removes duplicates based on position increment which wont work when using it after something like the edgengram filter. when i posted this to the mailing list even erik hatcher seemed to think thats what this filter was supposed to do...

attaching a patch that has the expected behaviour and initializes variables in constructor