From Piotr Idzikowski on java-user mailing list http://markmail.org/message/af26kr7fermt2tfh:
I am developing own analyzer based on StandardAnalyzer.
I realized that tokenizer.setMaxTokenLength is called many times.
Does it make sense if length stays the same? I see it finally calls this
one( in StandardTokenizerImpl ):
So it just copies old array content into the new one.