Bug 55524 - Deadlock produced during Websocket write operation (org.apache.catalina.websocket.WsOutbound)
Summary: Deadlock produced during Websocket write operation (org.apache.catalina.webso...
Status: RESOLVED FIXED
Alias: None
Product: Tomcat 7
Classification: Unclassified
Component: Servlet & JSP API (show other bugs)
Version: trunk
Hardware: PC Linux
: P2 blocker (vote)
Target Milestone: ---
Assignee: Tomcat Developers Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-09-04 23:34 UTC by confile
Modified: 2014-09-18 03:09 UTC (History)
0 users



Attachments
thread dump of the deadlock (160.57 KB, text/plain)
2013-09-04 23:34 UTC, confile
Details

Note You need to log in before you can comment on or make changes to this bug.
Description confile 2013-09-04 23:34:13 UTC
Created attachment 30800 [details]
thread dump of the deadlock

I use Tomcat with an application that uses Websockets. The websockets are handled by the Atmosphere framework: https://github.com/Atmosphere/atmosphere

A deadlock occurs during write operations to the websocket.

I have attached a thread dump of the deadlock. 

This issue is also duscussed here: 
https://github.com/Atmosphere/atmosphere/issues/1264


Here are the deadlocked threads: 


Found one Java-level deadlock:
=============================
"Atmosphere-Shared-AsyncOp-267":
  waiting to lock monitor 0x00007efebc0015f8 (object 0x00000005ef4c6988, a org.apache.catalina.websocket.WsOutbound),
  which is held by "Atmosphere-Scheduler-2"
"Atmosphere-Scheduler-2":
  waiting to lock monitor 0x00007efe8c290ac0 (object 0x00000005ef4b77f8, a org.atmosphere.cpr.AtmosphereResourceImpl),
  which is held by "Atmosphere-Shared-AsyncOp-267"

Java stack information for the threads listed above:
===================================================
"Atmosphere-Shared-AsyncOp-267":
	at org.apache.catalina.websocket.WsOutbound.writeTextMessage(WsOutbound.java:165)
	- waiting to lock <0x00000005ef4c6988> (a org.apache.catalina.websocket.WsOutbound)
	at org.atmosphere.container.version.TomcatWebSocket.write(TomcatWebSocket.java:49)
	at org.atmosphere.websocket.WebSocket.write(WebSocket.java:199)
	at org.atmosphere.websocket.WebSocket.write(WebSocket.java:168)
	at org.atmosphere.websocket.WebSocket.write(WebSocket.java:40)
	at org.atmosphere.cpr.AtmosphereResponse$2.write(AtmosphereResponse.java:574)
	at org.atmosphere.handler.AbstractReflectorAtmosphereHandler.onStateChange(AbstractReflectorAtmosphereHandler.java:169)
	at org.atmosphere.cpr.DefaultBroadcaster.invokeOnStateChange(DefaultBroadcaster.java:1027)
	at org.atmosphere.cpr.DefaultBroadcaster.prepareInvokeOnStateChange(DefaultBroadcaster.java:1047)
	at org.atmosphere.cpr.DefaultBroadcaster.executeAsyncWrite(DefaultBroadcaster.java:921)
	at org.atmosphere.cpr.DefaultBroadcaster$3.run(DefaultBroadcaster.java:580)
	- locked <0x00000005ef4b77f8> (a org.atmosphere.cpr.AtmosphereResourceImpl)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:724)
"Atmosphere-Scheduler-2":
	at org.atmosphere.cpr.AsynchronousProcessor.completeLifecycle(AsynchronousProcessor.java:476)
	- waiting to lock <0x00000005ef4b77f8> (a org.atmosphere.cpr.AtmosphereResourceImpl)
	at org.atmosphere.cpr.AsynchronousProcessor.timedout(AsynchronousProcessor.java:437)
	at org.atmosphere.cpr.AsynchronousProcessor$AsynchronousProcessorHook.timedOut(AsynchronousProcessor.java:633)
	at org.atmosphere.websocket.DefaultWebSocketProcessor.close(DefaultWebSocketProcessor.java:483)
	at org.atmosphere.container.TomcatWebSocketHandler.onClose(TomcatWebSocketHandler.java:80)
	at org.apache.catalina.websocket.StreamInbound.doOnClose(StreamInbound.java:222)
	at org.apache.catalina.websocket.WsOutbound.doWriteBytes(WsOutbound.java:423)
	at org.apache.catalina.websocket.WsOutbound.doWriteText(WsOutbound.java:442)
	at org.apache.catalina.websocket.WsOutbound.writeTextMessage(WsOutbound.java:174)
	- locked <0x00000005ef4c6988> (a org.apache.catalina.websocket.WsOutbound)
	at org.atmosphere.container.version.TomcatWebSocket.write(TomcatWebSocket.java:49)
	at org.atmosphere.websocket.WebSocket.write(WebSocket.java:199)
	at org.atmosphere.websocket.WebSocket.write(WebSocket.java:168)
	at org.atmosphere.websocket.WebSocket.write(WebSocket.java:40)
	at org.atmosphere.cpr.AtmosphereResponse$2.write(AtmosphereResponse.java:574)
	at org.atmosphere.cpr.AtmosphereResponse.write(AtmosphereResponse.java:992)
	at org.atmosphere.interceptor.HeartbeatInterceptor$1$1.call(HeartbeatInterceptor.java:104)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:724)
Comment 1 Mark Thomas 2013-09-05 10:47:47 UTC
I've only taken a quick look at this. So far, it appears that one lock is in Tomcat and one in Atmosphere so it is possible that changes will be required in both to fully fix this.

I'll see what I can to do narrow / remove the sync in Tomcat. I suspect narrowing is the more likely outcome.
Comment 2 Mark Thomas 2013-09-05 11:28:04 UTC
I believe I have fixed this in 7.0.x trunk and it will be included in 7.0.43 onwards.

If you are able to build Tomcat from source and confirm the fix that would be helpful.

There shouldn't be any need for changes in Atmosphere to fix this although a review of the relevant syncs in that code base as well wouldn't hurt.
Comment 3 confile 2013-09-05 11:49:04 UTC
Is this bug fixed in Tomcat8 as well?
Comment 4 Mark Thomas 2013-09-05 11:53:37 UTC
The bug was in Tomcat's proprietary WebSocket implementation. That implementation does not exist in Tomcat 8. It has been replaced by the JSR-356 implementation (which will also be in Tomcat 7 from the next release).
Comment 5 confile 2013-09-05 11:55:35 UTC
When will you release tomcat 7.0.43 and tomcat 8?
Comment 6 Christopher Schultz 2013-09-05 18:55:02 UTC
(In reply to confile from comment #5)
> When will you release tomcat 7.0.43 and tomcat 8?

When they are ready.
Comment 7 confile 2013-09-06 01:53:19 UTC
I tried to use the trunc version of Tomcat but this is even worse. I only get this message: 


Sep 06, 2013 3:52:14 AM org.apache.coyote.http11.AbstractHttp11Processor process
INFO: Error parsing HTTP request header
 Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level.
Comment 8 Mark Thomas 2013-09-06 15:32:32 UTC
Error parsing HTTP request header is a different problem so should be in a new bugzilla issue. Further there is insufficient information for any one to even begin to debug the problem you are seeing. You should seek help on the users list in the first instance. If that discussion concludes there is a Tomcat bug then at that point open a new issue and provide all the necessary information for the issue to be reproduced.
Comment 9 Dmitry Treskunov 2014-09-16 14:18:58 UTC
Looking at the changelog http://tomcat.apache.org/tomcat-7.0-doc/changelog.html I see that 

"Avoid a possible deadlock when one thread is shutting down a connection while another thread is trying to write to it. (markt)"

was included in 7.0.53, but not in 7.0.43

Am I right?
Comment 10 Violeta Georgieva 2014-09-18 03:09:51 UTC
(In reply to Dmitry Treskunov from comment #9)
> Looking at the changelog
> http://tomcat.apache.org/tomcat-7.0-doc/changelog.html I see that 
> 
> "Avoid a possible deadlock when one thread is shutting down a connection
> while another thread is trying to write to it. (markt)"
> 
> was included in 7.0.53, but not in 7.0.43
> 
> Am I right?

No

This fix was included in 7.0.43. See [1]

Regards,
Violeta

[1] 7.0.43 changelog - 55524: Refactor to avoid a possible deadlock when handling an IOException during output when using Tomcat' proprietary (and deprecated) WebSocket API. (markt)