[AXIS2-5453] Threads not getting closed at server side on High Load.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 1.5.5, 1.6.2
• Component/s: client-api
• Labels: adb-codegen client load
• Environment: OS: win2k R2 server; App server: apache-tomcat-7.0.32-windows-x64; Axis2 version:- Axis2 1.5.5 at server side AND Axis2 1.6.2 at client side.; Web service stubs generated using Axis2 1.6.2 jar files with ADB data binding.
• Assignee: Unassigned
• Reporter: AravindPS
• Created: 12/Nov/12 10:17
• Updated: 20/Dec/16 08:34
• Resolved: 19/Dec/16 14:38

Description

Hi,
We are doing a load test on web Services. When we run a test for 25 thread and 25 cycles per thread with each cycle processing a few web service requests, we are seeing that the number of threads for tomcat server is reaching the max limit of 300 and the threads are not closed automatically by client side or server side. The stubs are generated using the ADB databinding. We are using new instance of the stub for different threads. We are sending the requests from a single host.
Some config on client side include:
-------------------------------------------------------------------------------------------------------------------------
MultiThreadedHttpConnectionManager httpConnectionManager = new MultiThreadedHttpConnectionManager();
HttpClient httpClient = new HttpClient(httpConnectionManager);

stub._getServiceClient().getOptions().setProperty(HTTPConstants.REUSE_HTTP_CLIENT, Constants.VALUE_TRUE);

HttpConnectionManagerParams params = new HttpConnectionManagerParams();
params.setDefaultMaxConnectionsPerHost(100); /* 20 per host is fine in practical scenario */
params.setMaxTotalConnections(300);
httpConnectionManager.setParams(params);

stub._getServiceClient().getOptions().setProperty(HTTPConstants.CACHED_HTTP_CLIENT, httpClient);
stub._getServiceClient().getOptions().setProperty(HTTPConstants.SO_TIMEOUT, 10000);
stub._getServiceClient().getOptions().setProperty(HTTPConstants.CONNECTION_TIMEOUT, 10000);
stub._getServiceClient().getOptions().setProperty(HTTPConstants.CHUNKED, "false");

// to avoid retries
HttpMethodParams methodParams = new HttpMethodParams();
DefaultHttpMethodRetryHandler retryHandler = new DefaultHttpMethodRetryHandler(0, false);
methodParams.setParameter(HttpMethodParams.RETRY_HANDLER, retryHandler);
stub._getServiceClient().getOptions().setProperty(HTTPConstants.HTTP_METHOD_PARAMS, methodParams);
-------------------------------------------------------------------------------------------------------------------------

After running the threads I am calling:
-----------------------------------------------------------------------------
stub._getServiceClient().cleanupTransport();
stub._getServiceClient().cleanup();
stub.cleanup();
httpConnectionManager.closeIdleConnections(0);
httpConnectionManager.deleteClosedConnections();
httpConnectionManager.shutdown();
stub = null;
httpConnectionManager = null;
-----------------------------------------------------------------------------

After each web service call I am calling:
-----------------------------------------------------------------
stub._getServiceClient().cleanupTransport();
-----------------------------------------------------------------

When I shutdown the tomcat, I am getting 'severe' errors in the Tomcat log "catalina.2012-11-12.log". I have attached the log. All the threads are in "Waiting" state. I am not able to close the threads and server gets hanged and it cannot process any new requests.

Regards,
Aravind