Uploaded image for project: 'HttpComponents HttpClient'
  1. HttpComponents HttpClient
  2. HTTPCLIENT-1924

connection leak issue when OutOfMemory

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 4.5.3, 4.5.4, 4.5.5
    • Fix Version/s: 4.5.6, 5.0 Beta2
    • Component/s: HttpClient (classic)
    • Labels:
      None

      Description

      Yesterday, I found one of our java app is continuously throwing the following exception:

       

      _org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
      at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:313)
      at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:279)
      at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:191)
      at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
      at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
      at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
      at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
      at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
      at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)_

      at .....

       

      So I look into the log, I found that before this exception occurs, there are some `OutOfMemoryError`s (there are many different type of OutOfMemoryError, I just copy&paste the most interesting one): 

       

      java.lang.OutOfMemoryError: GC overhead limit exceeded
      at org.apache.http.util.CharArrayBuffer.substringTrimmed(CharArrayBuffer.java:466)
      at org.apache.http.message.BasicLineParser.parseProtocolVersion(BasicLineParser.java:163)
      at org.apache.http.message.BasicLineParser.parseStatusLine(BasicLineParser.java:366)
      at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:158)
      at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
      at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
      at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
      at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
      at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
      at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
      at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
      at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
      at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
      at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
      at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
      at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
      at ........
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)

       

      I look into the source code of httpClient, and found that when the `OutOfMemoryError` is occur at the exacting place, the `HttpConnection` will never be get released, so it will leak, and never to be reused any more, so the following http call will be rejected,  that's why `Timeout waiting for connection from pool` is *continuously trigged, even after my system have been recovered from the `GC` problem, this problem will never auto back to normal. so at this point to resolve this problem, my only choice is to fully restart my system.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              after_sss wenqi.huang
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: