[HTTPCLIENT-1918] CLOSE_WAIT issue in HttpClient

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 4.5.5
• Component/s: HttpClient (async)
• Assignee: Unassigned
• Reporter: Santhosh Sheshasayanan
• Created: 25/Apr/18 06:07
• Updated: 02/May/18 07:37
• Resolved: 01/May/18 07:35

Description

We are facing CLOSE_WAIT issue in our production system on heavy load. We are using Apache's httpClient 4.5.5 and httpcore 4.4.9 in our code. Sometime the server sends half of the data; sometime server sends no data and close the connection at their end. Now, httpclient puts the socket connection to CLOSE_WAIT indefinitely. 

We tried the following but nothing helped us.

• Set TTL to 1 minutes in PoolingHttpClientConnectionManager
• Used the following methods
  • evictExpiredConnections()
  • evictIdleConnections(5, TimeUnit.SECONDS)
  • https://hc.apache.org/httpcomponents-client-ga/tutorial/html/connmgmt.html
    • Topic "2.5. Connection eviction policy"
    • Topic "2.6. Connection keep alive strategy"

   Later, we found the method "setValidateAfterInactivity". Its description says "This check helps detect connections that have become stale (half-closed) while kept inactive in the pool". So we thought it will help us to detect and close the socket which are in "CLOSE_WAIT" state. But it did not help us.

Problem here, The HTTP client puts the socket into "CLOSE_WAIT" state and waiting for the signal from server to close the socket. But the server closed the connection already.  Please suggest us how to handle/resolve the issue?

 

Scala Code Snippet

private val clientConnectionManager =

{ val clientConnectionManager = new PoolingHttpClientConnectionManager(1, java.util.concurrent.TimeUnit.MINUTES); clientConnectionManager.setMaxTotal(cfg.connectionPoolSize) clientConnectionManager.setDefaultMaxPerRoute(cfg.connectionPoolSize) clientConnectionManager.setValidateAfterInactivity(30000); clientConnectionManager }

val monitorThread = new IdleConnectionMonitorThread(clientConnectionManager);
monitorThread.start();