spinoff from user-list thread.
I think the rename helps, but the javadocs still have problems: they seem to only describe a totally wacky case (CachingTokenFilter) and not the normal case.
Ideally setReader would be final I think, but there are a few crazy tokenstreams to fix before I could make that work. Would also need something hackish so MockTokenizer's state machine is still functional.
But i worked on fixing up the mess in our various tokenstreams, which is easy for the most part.
As part of this I found it was really useful in flushing out test bugs (ones that dont use MockTokenizer, which they really should), if we can do some best-effort exceptions when the consumer is broken and it costs nothing.
I think this is worth exploring more... this was really effective at finding broken tests etc. We should see if we can be more thorough/ideally throw better exceptions when consumers are broken and its free.