Bug 55241

Summary:	maxConnections for BIO connector leads to acceptor stop
Product:	Tomcat 7	Reporter:	kudryashov
Component:	Connectors	Assignee:	Tomcat Developers Mailing List <dev>
Status:	RESOLVED INVALID
Severity:	normal	CC:	vyacheslav.gusev
Priority:	P2
Version:	7.0.42
Target Milestone:	---
Hardware:	PC
OS:	Linux
Attachments:	tcp dump for reproduction Thread stack of hanged tomcat connector

Description kudryashov 2013-07-11 13:10:13 UTC

It looks like issue #53173 is not fixed completely. Recently we had a similar issue under Tomcat 7.0.34 / BIO connector when Acceptor thread hung forever on LimitLatch:

"http-bio-63.215.91.155-8443-Acceptor-0" daemon prio=10 tid=0x0000000001656800 nid=0x540c waiting on condition [0x00007fb64aa2f000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x0000000763e49970> (a org.apache.tomcat.util.threads.LimitLatch$Sync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
	at org.apache.tomcat.util.threads.LimitLatch.countUpOrAwait(LimitLatch.java:115)
	at org.apache.tomcat.util.net.AbstractEndpoint.countUpOrAwaitConnection(AbstractEndpoint.java:718)
	at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:210)
	at java.lang.Thread.run(Thread.java:722)

Tomcat is run with maxThreads="1000" and maxConnections="1000"
This can be reproduced via the following scenario in our environment:
1. Start tcpkill in the following way to kill the connection from client to Tomcat for requests larger than 2000:
tcpkill host <host> and port 8185 and greater 2000
2. Access Tomcat several times. As soon as the number of ClientAbortException errors reaches the number of threads (or connections), Acceptor thread blocks in WAITING state and Tomcat stops accepting new connections

Comment 1 Mark Thomas 2013-07-11 20:48:15 UTC

I have tested this with the latest Tomcat 7.0.x code and with trunk on both Windows and Linux and am unable to reproduce it.

If you still see this issue with the latest 7.0.x release feel free to re-open this issue but you'll need to add enough information for the issue to be reproduced.

What might help is if you have a stack trace of the ClientAbortException as reported by Tomcat as it might show a code path where the socket isn't getting removed from the count once it closes.

Comment 2 vyacheslav 2013-07-23 18:35:52 UTC

Created attachment 30619 [details]
tcp dump for reproduction

Reproduced on Tomcat/7.0.42

8185 - is port of tomcat.
Connection is established and in the middle of communication packet len=7300 is sent from tomcat to client. This packet is blocked by tcpkill expression "host 89.113.129.49 and port 8185 and greater 2000". This blocking action are creating ClientAbortException.

Comment 3 vyacheslav 2013-07-23 18:38:36 UTC

For easier reproduce we use 5 threads in executor. Here head of our server.xml

<Server port="${SHUTDOWN_PORT}" shutdown="SHUTDOWN">

  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"/>

  <GlobalNamingResources/>

  <Service name="${target}">

     <Executor name="common_executor"
               maxThreads="5"
		minSpareThreads="1"
     />

     <Connector port="${HTTPS_PORT}"
               address="${HTTP_ADDR}"
               executor="common_executor"
               enableLookups="false"
               disableUploadTimeout="true"
               acceptCount="5"
               connectionTimeout="10000"
               keepAliveTimeout="0"
               maxKeepAliveRequests="-1"
	       scheme="https" secure="true"
               clientAuth="false" SSLEnabled="true" sslProtocol="TLS"
               keystorePass="keystorepass"
               keystoreFile="keystore.p12"
               keystoreType="PKCS12"
    />



After 5 ClientAbortExeptions we stuck.

Comment 4 Mark Thomas 2013-07-23 19:35:14 UTC

Re-open to see if there is enough information to reproduce this. The stacktrace from a ClientAbortException may be helpful.

Note: Using SSL & Executor

Comment 5 vyacheslav 2013-07-24 10:44:38 UTC

ClientAbortException:  java.net.SocketException: Connection reset
	at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:406)
	at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480)
	at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:310)
	at org.apache.catalina.connector.OutputBuffer.writeByte(OutputBuffer.java:450)
	at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:77)
	at java.io.DataOutputStream.writeInt(DataOutputStream.java:200)
	at gft.chart.impl.QuotesResponse.storeObject(QuotesResponse.java:38)
	at common.util.storable.StorableOutputStream.writeStorableCursor(StorableOutputStream.java:117)
	at common.util.storable.StorableOutputStream.writeSingle(StorableOutputStream.java:71)
	at common.util.storable.StorableOutputStream.writeObject(StorableOutputStream.java:58)
	at common.util.rif.RequestResponseSerializer.serializeResponse(RequestResponseSerializer.java:88)
	at common.gateway.HttpGateway.finalizeRequestProcessing(HttpGateway.java:494)
	at common.gateway.HttpGateway.processRemoteInvocation(HttpGateway.java:440)
	at common.gateway.HttpGateway.doPost(HttpGateway.java:154)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1023)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:722)
Caused by: java.net.SocketException: Connection reset
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
	at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:377)
	at sun.security.ssl.OutputRecord.write(OutputRecord.java:363)
	at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:830)
	at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:801)
	at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:122)
	at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:215)
	at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480)
	at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:366)
	at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:240)
	at org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:84)
	at org.apache.coyote.http11.AbstractOutputBuffer.doWrite(AbstractOutputBuffer.java:192)
	at org.apache.coyote.Response.doWrite(Response.java:517)
	at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:401)
	... 30 more
Exception in thread "tomcat-exec-7" java.lang.InternalError: response serialization failed: null
	at common.util.rif.RequestResponseSerializer.serializeResponse(RequestResponseSerializer.java:91)
	at common.gateway.HttpGateway.finalizeRequestProcessing(HttpGateway.java:494)
	at common.gateway.HttpGateway.processRemoteInvocation(HttpGateway.java:440)
	at common.gateway.HttpGateway.doPost(HttpGateway.java:154)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1023)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:722)

Comment 6 vyacheslav 2013-07-24 14:33:53 UTC

Reproduced also with APR. NOT reproduced with NIO.

Here is APR ClientAbortException stack trace.




ClientAbortException:  java.io.IOException
        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:388)
        at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:462)
        at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:310)
        at org.apache.catalina.connector.OutputBuffer.writeByte(OutputBuffer.java:432)
        at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:77)
        at java.io.DataOutputStream.writeInt(DataOutputStream.java:200)
        at gft.chart.impl.QuotesResponse.storeObject(QuotesResponse.java:38)
        at common.util.storable.StorableOutputStream.writeStorableCursor(StorableOutputStream.java:117)
        at common.util.storable.StorableOutputStream.writeSingle(StorableOutputStream.java:71)
        at common.util.storable.StorableOutputStream.writeObject(StorableOutputStream.java:58)
        at common.util.rif.RequestResponseSerializer.serializeResponse(RequestResponseSerializer.java:88)
        at common.gateway.HttpGateway.finalizeRequestProcessing(HttpGateway.java:494)
        at common.gateway.HttpGateway.processRemoteInvocation(HttpGateway.java:440)
        at common.gateway.HttpGateway.doPost(HttpGateway.java:154)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
        at org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run(AprEndpoint.java:1780)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.IOException
        at org.apache.coyote.http11.InternalAprOutputBuffer.flushBuffer(InternalAprOutputBuffer.java:205)
        at org.apache.coyote.http11.InternalAprOutputBuffer.access$100(InternalAprOutputBuffer.java:37)
        at org.apache.coyote.http11.InternalAprOutputBuffer$SocketOutputBuffer.doWrite(InternalAprOutputBuffer.java:235)
        at org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:84)
        at org.apache.coyote.http11.AbstractOutputBuffer.doWrite(AbstractOutputBuffer.java:192)
        at org.apache.coyote.Response.doWrite(Response.java:505)
        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:383)
        ... 30 more
Exception in thread "tomcat-exec-5" java.lang.InternalError: response serialization failed: null
        at common.util.rif.RequestResponseSerializer.serializeResponse(RequestResponseSerializer.java:91)
        at common.gateway.HttpGateway.finalizeRequestProcessing(HttpGateway.java:494)
        at common.gateway.HttpGateway.processRemoteInvocation(HttpGateway.java:440)
        at common.gateway.HttpGateway.doPost(HttpGateway.java:154)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
        at org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run(AprEndpoint.java:1780)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)

Comment 7 Mark Thomas 2013-07-26 13:39:16 UTC

One last piece, please provide a full thread dump from the Tomcat instance once the acceptor has stopped accepting new connections. A thread dump from when you are using just 5 threads in the executor is fine - in fact it is preferred.

Comment 8 vyacheslav 2013-07-26 18:56:36 UTC

Created attachment 30634 [details]
Thread stack of hanged tomcat connector

Comment 9 Mark Thomas 2013-07-30 08:20:47 UTC

Thanks for the additional information. This does point towards a Tomcat bug. I'm currently lookng at the stack trace to see if that provides any clues as to what is going wrong.

Comment 10 Mark Thomas 2013-07-30 09:05:06 UTC

Your code is throwing java.lang.InternalError which is a sub-class of java.lang.VirtualMachineError

The javadoc states that VirtualMachineError is:
<quote>
Thrown to indicate that the Java Virtual Machine is broken or has run out of resources necessary for it to continue operating.
</quote>

Tomcat handles application code throwing any Throwablte with two exceptions:
- OutOfMemoryError
- VirtualMachineError
as both of these are considered fatal to the JVM.

In this case the application should not be throwing java.lang.InternalError as the JVM is neither broken nor has it run out of resources.

The reason I could not repeat this is that your steps to reproduce generates a ClientAbortException which is correctly handled. My test did not generate a java.lang.InternalError

That NIO did not exhibit the problem was due to a bug in the NIO connector (now fixed) that incorrectly swallowed VirtualMachineErrors.