Major Description
The javadocs claim that too-long tokens are discarded, but in fact they are simply chopped up. The following test case unexpectedly passes:
public void testMaxTokenLengthNonDefault() throws Exception {
StandardAnalyzer a = new StandardAnalyzer();
a.setMaxTokenLength(5);
assertAnalyzesTo(a, "ab cd toolong xy z", new String[]{"ab", "cd", "toolo", "ng", "xy", "z"});
a.close();
}
We should at least fix the javadocs ...
(I hit this because I was trying to also add setMaxTokenLength to EnglishAnalyzer).