[HTTPCORE-130] Potential for deadlock while using a ThrottlingHttpServiceHandler and a SSLIOSession

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0-alpha6
• Fix Version/s: 4.0-beta1
• Component/s: HttpCore
• Environment: Linux RHEL 4, jdk 1.6_02
• Assignee: Oleg Kalnichevski
• Reporter: Lorenzo Moretti
• Created: 22/Nov/07 12:19
• Updated: 09/Feb/09 21:38
• Resolved: 23/Nov/07 08:42

Description

The following message was originally posted under HTTPCORE-48 but Oleg asked to created a new issue for it.

I have been working with HttpCore alpha 6 for the past weeks and recently ran into a deadlock. Since I am quite new to both NIO and SSL it might very well be that the code for my tests is inappropriate. I obtained the following deadlock stack trace using "kill -QUIT" on the process.

After the deadlock description I have very briefly explained what my test methods do. I am not sure they are relevant since they do not do any locking themselves, but that in itself might be the source of the problem!

Object <0x8c65a988>, if I am not mistaken, is the SharedInputBuffer.mutex.
Thread "RequestWorker_12" was spawned within ThrottlingHttpServiceHandler.handleRequest() by this.executor.execute().
Thread "TestService_IOD_11" was spawned within AbstractMultiworkerIOReactor.execute() by this.threadFactory.newThread()

=============================
"RequestWorker_12":
waiting to lock monitor 0x08083ac8 (object 0x8c61cd18, a org.apache.http.impl.nio.reactor.SSLIOSession),
which is held by "TestService_IOD_11"
"TestService_IOD_11":
waiting to lock monitor 0x08083a64 (object 0x8c65a988, a java.lang.Object),
which is held by "RequestWorker_12"

Java stack information for the threads listed above:
===================================================
"RequestWorker_12":
at org.apache.http.impl.nio.reactor.SSLIOSession.setEvent(SSLIOSession.java:371)

• waiting to lock <0x8c61cd18> (a org.apache.http.impl.nio.reactor.SSLIOSession)
  at org.apache.http.impl.nio.NHttpConnectionBase.requestInput(NHttpConnectionBase.java:154)
  at org.apache.http.nio.util.SharedInputBuffer.waitForData(SharedInputBuffer.java:104)
• locked <0x8c65a988> (a java.lang.Object)
  at org.apache.http.nio.util.SharedInputBuffer.read(SharedInputBuffer.java:137)
• locked <0x8c65a988> (a java.lang.Object)
  at org.apache.http.nio.entity.ContentInputStream.read(ContentInputStream.java:63)
  at TestHttpServer.readData(TestHttpServer.java:451)
  at TestHttpServer.access$200(TestHttpServer.java:62)
  at TestHttpServer$TestHandler3.handle(TestHttpServer.java:312)
  at org.apache.http.nio.protocol.ThrottlingHttpServiceHandler.handleRequest(ThrottlingHttpServiceHandler.java:477)
  at org.apache.http.nio.protocol.ThrottlingHttpServiceHandler.access$000(ThrottlingHttpServiceHandler.java:91)
  at org.apache.http.nio.protocol.ThrottlingHttpServiceHandler$1.run(ThrottlingHttpServiceHandler.java:195)
  at java.lang.Thread.run(Thread.java:619)
  at DefaultRequestExecutor$Worker.run(DefaultRequestExecutor.java:43)
  "TestService_IOD_11":
  at org.apache.http.nio.util.SharedInputBuffer.consumeContent(SharedInputBuffer.java:71)
• waiting to lock <0x8c65a988> (a java.lang.Object)
  at org.apache.http.nio.protocol.ThrottlingHttpServiceHandler.inputReady(ThrottlingHttpServiceHandler.java:227)
• locked <0x8c658408> (a org.apache.http.nio.protocol.ThrottlingHttpServiceHandler$ServerConnState)
  at org.apache.http.impl.nio.DefaultNHttpServerConnection.consumeInput(DefaultNHttpServerConnection.java:135)
  at org.apache.http.impl.nio.reactor.SSLServerIOEventDispatch.inputReady(SSLServerIOEventDispatch.java:132)
• locked <0x8c61cd18> (a org.apache.http.impl.nio.reactor.SSLIOSession)
  at IoEventDispatchWrapper.inputReady(IoEventDispatchWrapper.java:45)
  at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:137)
  at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:141)
  at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:69)
  at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:281)
  at DebugThreadFactory$1.run(DebugThreadFactory.java:29)
  at java.lang.Thread.run(Thread.java:619)

Found 1 deadlock.

RequestWorker_12 uses:

• DefaultRequestExecutor$Worker.run(): runs the Runnable created by ThrottlingHttpServiceHandler.requestReceived().
• TestHttpServer$TestHandler3.handle(): acts as an echo by setting the response body with the contents of the request body (status is set to SC_OK).
• TestHttpServer.readData(): transfers bytes from an HttpEntityEnclosingRequest.getEntity().getContent() to a ByteArrayOutputStream.

DebugThreadFactory$1is a dynamically created Thread based on the Runnable passed to ThreadFactory.newThread().

TestService_IOD_11 uses:

• DebugThreadFactory$1.run(): sets logging information before invoking Runnable.run().
• IoEventDispatchWrapper.inputReady(): invokes the IOEventDispatch.inputReady() for the dispatch it wraps.

Thanks for your time,

– Lorenzo