Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
4.5.3, 4.5.4, 4.5.5
-
None
Description
Yesterday, I found one of our java app is continuously throwing the following exception:
_org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:313)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:279)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:191)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)_
at .....
So I look into the log, I found that before this exception occurs, there are some `OutOfMemoryError`s (there are many different type of OutOfMemoryError, I just copy&paste the most interesting one):
java.lang.OutOfMemoryError: GC overhead limit exceeded
at org.apache.http.util.CharArrayBuffer.substringTrimmed(CharArrayBuffer.java:466)
at org.apache.http.message.BasicLineParser.parseProtocolVersion(BasicLineParser.java:163)
at org.apache.http.message.BasicLineParser.parseStatusLine(BasicLineParser.java:366)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:158)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at ........
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
I look into the source code of httpClient, and found that when the `OutOfMemoryError` is occur at the exacting place, the `HttpConnection` will never be get released, so it will leak, and never to be reused any more, so the following http call will be rejected, that's why `Timeout waiting for connection from pool` is *continuously trigged, even after my system have been recovered from the `GC` problem, this problem will never auto back to normal. so at this point to resolve this problem, my only choice is to fully restart my system.*
Attachments
Attachments
Issue Links
- causes
-
HTTPCORE-735 "Connection pool shut down" unreasonable
- Resolved