I ran into a deadlock in the MutextTransport.oneway(Command command) function when processing very high message volume (100% cpu utilization) at the time of a shutdown. I'm running 4.0.2 on WinXP and within a demand forwarding environment with failover = true.
I did trap this deadlock in the debugger and it looks like two commands are crossing paths in opposite directions through the MutexTransport. One of the commands is a MessageDispatch and the other is a ShutdownInfo. Now, when the ShutdownInfo gets through the MutexTransport first, it tries to shutdown the background thread of the TcpTransport. However, this thread is currently servicing the MessageDispatch and is blocked on the MutexTransport. Deadlock.
So, my patch was simply to avoid entering the synchronized(writeMutex) block in the oneway(Command command) function of MutexTransport if
command.isShutdownInfo() returns true:
if (command.isShutdownInfo())
next.oneway(command);
else
synchronized(writeMutex) {
next.oneway(command);
}
Bug
Resolved
Major
Deadlock in MutexTransport on shutdown with high volume of messages