The problem seems to relate to parsing the body content when the **Transfer- Encoding: chunked** header is present. It appears that the chunk length gets corrupted under load and the client is unable to parse the chunk out of the response body. When the NIO connector parameter **socket.appWriteBufSize** is set to a value larger than the total response body, the error condition does not occur. One might succesfully argue that this is proper performance tuning. The Tomcat documents point out that to scale a large number of long held connections, the buffer sizes may need to be less than the response body (the memory footprint would be quite large otherwise). Could there be a race condition involving the response buffering code? I have confirmed this behavior on both JDKs 1.5 and 1.6 on both Windows 2003 and Linux. Apache Bench does not appear to fully parse the response so it will not be helpful in reproducing the issue. The Grinder framework does. I am including the Grinder scripts so that the issue may be reproduced.
Created attachment 21114 [details] Jython/Grinder script Jython script referenced from grinder properties file. This script does most of the interaction with the Tomcat server. It is configured via a grinder properties file.
Created attachment 21115 [details] Grinder properties file Use this file as a starting point for your Grinder setup. This defines the number of threads, etc. to configure the test client.
Created attachment 21116 [details] Reponse generator servlet This is a very simple Java servlet. It generates a reponse body >8K. This somewhat larger response seems to render with Transfer-Encoding: chunked rather than a Content-Length header. This is what we want because the smaller response bodies seem to work OK. In the real world responses larger than 8K are the norm.
Thank you, I will take a look. Filip
I'm unable to get your grinder scripts to work. There are several errors that happen, for one, random is not available on a clean download of grinder. what would be helpful, is if you could create a test case, that I can run without trying to figure out grinder (sorry I'm a newbie to this) for most of a day. thanks Filip
Created attachment 21123 [details] Use this Jython script instead Use this script instead of the first one attached. Sorry for the trouble Filip.
My apolgies Filip. I've attached an updated Jython script. This one does not depend on random.
I'm able to reproduce the problem in the connector using a JMeter script. The error doesn't exist in the previous trunk connector, now in sandbox http://svn.apache.org/viewvc/tomcat/sandbox/gdev6x/ There are features in that branch not present in current release. I will look into the option of porting the NIO enhancements from that branch to 6.0.x Filip
I've created a patch for 6.0.x branch (a port from the sandbox) http://people.apache.org/~fhanik/patches/apache-tomcat-6.0.x-niofix.zip feel free to try it out, and let me know the feedback Filip
I've run a series of tests of various kinds and it appears that this issue is fixed in the patched version. Would this fix going into a 6.0.15 release?
Patch has been added to 6.0.x and will go out with 6.0.16 next release. Thanks for the feedback.