Created attachment 30800 [details] thread dump of the deadlock I use Tomcat with an application that uses Websockets. The websockets are handled by the Atmosphere framework: https://github.com/Atmosphere/atmosphere A deadlock occurs during write operations to the websocket. I have attached a thread dump of the deadlock. This issue is also duscussed here: https://github.com/Atmosphere/atmosphere/issues/1264 Here are the deadlocked threads: Found one Java-level deadlock: ============================= "Atmosphere-Shared-AsyncOp-267": waiting to lock monitor 0x00007efebc0015f8 (object 0x00000005ef4c6988, a org.apache.catalina.websocket.WsOutbound), which is held by "Atmosphere-Scheduler-2" "Atmosphere-Scheduler-2": waiting to lock monitor 0x00007efe8c290ac0 (object 0x00000005ef4b77f8, a org.atmosphere.cpr.AtmosphereResourceImpl), which is held by "Atmosphere-Shared-AsyncOp-267" Java stack information for the threads listed above: =================================================== "Atmosphere-Shared-AsyncOp-267": at org.apache.catalina.websocket.WsOutbound.writeTextMessage(WsOutbound.java:165) - waiting to lock <0x00000005ef4c6988> (a org.apache.catalina.websocket.WsOutbound) at org.atmosphere.container.version.TomcatWebSocket.write(TomcatWebSocket.java:49) at org.atmosphere.websocket.WebSocket.write(WebSocket.java:199) at org.atmosphere.websocket.WebSocket.write(WebSocket.java:168) at org.atmosphere.websocket.WebSocket.write(WebSocket.java:40) at org.atmosphere.cpr.AtmosphereResponse$2.write(AtmosphereResponse.java:574) at org.atmosphere.handler.AbstractReflectorAtmosphereHandler.onStateChange(AbstractReflectorAtmosphereHandler.java:169) at org.atmosphere.cpr.DefaultBroadcaster.invokeOnStateChange(DefaultBroadcaster.java:1027) at org.atmosphere.cpr.DefaultBroadcaster.prepareInvokeOnStateChange(DefaultBroadcaster.java:1047) at org.atmosphere.cpr.DefaultBroadcaster.executeAsyncWrite(DefaultBroadcaster.java:921) at org.atmosphere.cpr.DefaultBroadcaster$3.run(DefaultBroadcaster.java:580) - locked <0x00000005ef4b77f8> (a org.atmosphere.cpr.AtmosphereResourceImpl) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) "Atmosphere-Scheduler-2": at org.atmosphere.cpr.AsynchronousProcessor.completeLifecycle(AsynchronousProcessor.java:476) - waiting to lock <0x00000005ef4b77f8> (a org.atmosphere.cpr.AtmosphereResourceImpl) at org.atmosphere.cpr.AsynchronousProcessor.timedout(AsynchronousProcessor.java:437) at org.atmosphere.cpr.AsynchronousProcessor$AsynchronousProcessorHook.timedOut(AsynchronousProcessor.java:633) at org.atmosphere.websocket.DefaultWebSocketProcessor.close(DefaultWebSocketProcessor.java:483) at org.atmosphere.container.TomcatWebSocketHandler.onClose(TomcatWebSocketHandler.java:80) at org.apache.catalina.websocket.StreamInbound.doOnClose(StreamInbound.java:222) at org.apache.catalina.websocket.WsOutbound.doWriteBytes(WsOutbound.java:423) at org.apache.catalina.websocket.WsOutbound.doWriteText(WsOutbound.java:442) at org.apache.catalina.websocket.WsOutbound.writeTextMessage(WsOutbound.java:174) - locked <0x00000005ef4c6988> (a org.apache.catalina.websocket.WsOutbound) at org.atmosphere.container.version.TomcatWebSocket.write(TomcatWebSocket.java:49) at org.atmosphere.websocket.WebSocket.write(WebSocket.java:199) at org.atmosphere.websocket.WebSocket.write(WebSocket.java:168) at org.atmosphere.websocket.WebSocket.write(WebSocket.java:40) at org.atmosphere.cpr.AtmosphereResponse$2.write(AtmosphereResponse.java:574) at org.atmosphere.cpr.AtmosphereResponse.write(AtmosphereResponse.java:992) at org.atmosphere.interceptor.HeartbeatInterceptor$1$1.call(HeartbeatInterceptor.java:104) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724)
I've only taken a quick look at this. So far, it appears that one lock is in Tomcat and one in Atmosphere so it is possible that changes will be required in both to fully fix this. I'll see what I can to do narrow / remove the sync in Tomcat. I suspect narrowing is the more likely outcome.
I believe I have fixed this in 7.0.x trunk and it will be included in 7.0.43 onwards. If you are able to build Tomcat from source and confirm the fix that would be helpful. There shouldn't be any need for changes in Atmosphere to fix this although a review of the relevant syncs in that code base as well wouldn't hurt.
Is this bug fixed in Tomcat8 as well?
The bug was in Tomcat's proprietary WebSocket implementation. That implementation does not exist in Tomcat 8. It has been replaced by the JSR-356 implementation (which will also be in Tomcat 7 from the next release).
When will you release tomcat 7.0.43 and tomcat 8?
(In reply to confile from comment #5) > When will you release tomcat 7.0.43 and tomcat 8? When they are ready.
I tried to use the trunc version of Tomcat but this is even worse. I only get this message: Sep 06, 2013 3:52:14 AM org.apache.coyote.http11.AbstractHttp11Processor process INFO: Error parsing HTTP request header Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level.
Error parsing HTTP request header is a different problem so should be in a new bugzilla issue. Further there is insufficient information for any one to even begin to debug the problem you are seeing. You should seek help on the users list in the first instance. If that discussion concludes there is a Tomcat bug then at that point open a new issue and provide all the necessary information for the issue to be reproduced.
Looking at the changelog http://tomcat.apache.org/tomcat-7.0-doc/changelog.html I see that "Avoid a possible deadlock when one thread is shutting down a connection while another thread is trying to write to it. (markt)" was included in 7.0.53, but not in 7.0.43 Am I right?
(In reply to Dmitry Treskunov from comment #9) > Looking at the changelog > http://tomcat.apache.org/tomcat-7.0-doc/changelog.html I see that > > "Avoid a possible deadlock when one thread is shutting down a connection > while another thread is trying to write to it. (markt)" > > was included in 7.0.53, but not in 7.0.43 > > Am I right? No This fix was included in 7.0.43. See [1] Regards, Violeta [1] 7.0.43 changelog - 55524: Refactor to avoid a possible deadlock when handling an IOException during output when using Tomcat' proprietary (and deprecated) WebSocket API. (markt)