[HTTPCLIENT-1924] connection leak issue when OutOfMemory

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.5.3, 4.5.4, 4.5.5
• Fix Version/s: 4.5.6, 5.0 Beta2
• Component/s: HttpClient (classic)
• Assignee: Unassigned
• Reporter: wenqi.huang
• Created: 11/Jun/18 06:08
• Updated: 27/Feb/23 05:11
• Resolved: 17/Jun/18 08:57

Description

Yesterday, I found one of our java app is continuously throwing the following exception:

 

_org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:313)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:279)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:191)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)_

at .....

 

So I look into the log, I found that before this exception occurs, there are some `OutOfMemoryError`s (there are many different type of OutOfMemoryError, I just copy&paste the most interesting one): 

 

java.lang.OutOfMemoryError: GC overhead limit exceeded
at org.apache.http.util.CharArrayBuffer.substringTrimmed(CharArrayBuffer.java:466)
at org.apache.http.message.BasicLineParser.parseProtocolVersion(BasicLineParser.java:163)
at org.apache.http.message.BasicLineParser.parseStatusLine(BasicLineParser.java:366)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:158)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at ........
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

 

I look into the source code of httpClient, and found that when the `OutOfMemoryError` is occur at the exacting place, the `HttpConnection` will never be get released, so it will leak, and never to be reused any more, so the following http call will be rejected,  that's why `Timeout waiting for connection from pool` is *continuously trigged, even after my system have been recovered from the `GC` problem, this problem will never auto back to normal. so at this point to resolve this problem, my only choice is to fully restart my system.* 

Issue Links

• causes: HTTPCORE-735 "Connection pool shut down" unreasonable