Major Description
I recently noticed a situation in which my Query analyzer was producing the same Token more then once, resulting in it getting two equally boosted clauses in the resulting query. In my specific case, i was using the same synonym file for multiple fields (some stemmed some not) and two synonyms for a word stemmed to the same root, which ment that particular word was worth twice as as any of the other variations of the synonym – but I can imagine other situations where this might come up, both at index time and at query time, particularlay when using SynonymFilter in combination with the WordDelimiter filter.
It occured to me that a DeDupFilter would be handy. In it's simplest form it would drop any Token it gets where the startOffset, endOffset,termText,and type are all identical to the previous token and the positionIncriment is 0. A more robust implimentation might support init options indicating that only certain combinations of those things should be used to determine equality (ie: just termText, just termText and positionIncriment=0, etc...) but in this case, an option might also be neccessary to determine with of the Tokens should be propogated (the first of the last)
