Since this path is a little complex, let me give some some explains.
The main modification is in Scanner.expandBuffer(), originally it simply uses a new larger char buffer to replace the old one, and copy all from the old one, so all properties (charbuffer length, capacity, limit and Scanner's findStartIndex, preStartIndex, cacheHasNextIndex and etc. )related to the old buffer won't change.
However in this patch, Scanner will try to discard the contends which are not used any longer in the current char buffer(which also may be replaced by a larger one) and only keep that will used for next Scanner.hasNext() or Scanner.next(), so the properties related to the char buffer also need be changed.
The next point is this patch add 2 instance variables tokenStart and tokenEnd which were just 2 local vars in Scanner.setTokenRegion(), since the char buffer's properties now may be changed at any time, now methods(like findPreDelimiter() and findPostDelimiter() etc.) which track the start index and end index need change this 2 vars at any moment.
if any one would like to try this patch? Thanks a lot!