Major Description
I've discovered an issue with the SslHandler class when the unwrap method is called on the local SSLEngine member (SslHandler.sslEngine).
If the returned status is SSLEngineResult.Status.BUFFER_OVERFLOW, the capacity of the output buffer (SslHandler.appBuffer) can be increased to a size which still may is not large enough for the result.
I have reproduced this issue consistently by sending a 4k frame over TLS1.2 to an android device. The frame gets heavily fragmented, sometimes into 6 frames, and the SSLEngine does not unwrap the frame until all the bytes have been received (since the hash is based on the entire frame).
Since the frame gets heavily fragmented, the last segment of the frame can be lower than 2048 bytes. Hence by increasing the capacity by << 1, the output buffer will still be under the required size. (Have a look through SslHandler source for "appBuffer.capacity(appBuffer.capacity() << 1);")
Anyway, the fix is really easy. Change the line:
appBuffer.capacity(appBuffer.capacity() << 1);
to:
appBuffer.capacity(sslEngine.getSession().getApplicationBufferSize());
This is actually in the java docs (http://docs.oracle.com/javase/7/docs/api/javax/net/ssl/SSLEngine.html) for the overflow buffer case.
Hope this helps,
Terence
