Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Fixed
-
1.1.1
-
None
-
Windows (XP, Vista) and Linux (Red-Hat Enterprise 5 and Ubuntu 7.10, 8.04) platforms, performance tuned and not.
Description
I can consistently reproduce an HTTP-NIO lockup by sending larger messages (> 300KB - both with and without attachments) and >50 threads of concurrency using SoapUI as a client. I'm directing the messages through a simple forwarding proxy service. This happens for me on both Windows and Linux (Red-Hat and Ubuntu) platforms, both tuned and not.
After a lot of investigation, here's what I've found:
- each of the transport receiver I/O dispatcher threads can block writing to the request pipe sink in ServerHandler's inputReady() method when there are no ServerWorker processing threads left
- the block occurs because the pipe's sink can only buffer a limited number of bytes until a ServerWorker thread is actively reading from the pipe's source
- a blocked I/O dispatcher thread stops all incoming reads from the client and writes back to the client for its associated connections
- as more requests come in with no free ServerWorker threads, more of the incoming I/O dispatcher threads are blocked until they are all permanently blocked
- the ServerWorker threads are all blocked either waiting for a free ClientWorker thread or blocked waiting for more input from the client (because the incoming mediation can be complete before a request has been fully read from the incoming socket - this is where I think the larger messages come into play)
- the ClientWorker threads are all busy waiting to send to their responses back to the client (as previously mentioned, the socket writes back to the client have been for all I/O dispatcher threads)
- there's no way out of the situation, so Synapse is effectively disabled. As you can see, increasing the number of I/O dispatchers and worker threads can only delay and not fix the problem.
Since ClientHandler's inputReady() can also block in this way, it probably should be fixed also.