Service mix hangs when the # concurrent requests exceeds the maxConnectionsPerHost

• Key: SM-1002
• Type: Bug
• Status: Resolved
• Resolution: Fixed
• Priority: Major
• Assignee: Guillaume Nodet
• Reporter: Ruwan Linton
• Component/s: servicemix-http
• Fix Version/s: 3.2

Description

When I am doing a performance test I have sent more than 40 concurrent requests from a single host to servicemix, and the maxConnectionsPerHost was 40. It is acceptable the failing to respond when it exceeds 40 (maxConnectionsPerHost) concurrent requests.

But afterward, SM hangs.

Comments

[Thomas Termin]

Can you give a bit more information. For example describe the involved components. Could you provide a threaddump for it? (kill -3)

[Thomas Termin]

Maybe you could add an example which I could try.

[Ruwan Linton]

I just deployed a http consumer and a provider linked directly to create a proxy for an existing service

xbean config for the http-consumer

<?xml version="1.0"?>
<beans xmlns:http="http://servicemix.apache.org/http/1.0"
xmlns:b="http://esb.wso2.org/perf/proxy">

<http:endpoint service="b:http"
endpoint="endpoint"
targetService="b:SimpleStockQuoteService"
role="consumer"
locationURI="http://10.100.1.43:8192/proxy"
defaultMep="http://www.w3.org/2004/08/wsdl/in-out"
soap="true"/>
</beans>

and here is the xbean config for the provider

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns:http="http://servicemix.apache.org/http/1.0"
xmlns:b="http://esb.wso2.org/perf/proxy">

<http:endpoint service="b:SimpleStockQuoteService"
endpoint="soap"
role="provider"
locationURI="http://10.100.1.44:9000/soap/SimpleStockQuoteService"
defaultMep="http://www.w3.org/2004/08/wsdl/in-out"
soap="true"
soapVersion="1.1"/>

</beans>

and the component.properties file entry servicemix-http.maxConnectionsPerHost was set to 30 and after 36 requests it fails and hangs,

To get more information about our perf test please refer to the article http://wso2.org/library/1721

[Bruce Snyder]

Ruwan, can you provide the SimpleStockQuoteService WAR for testing, or maybe even just the source for the servlet?

The source code for this client and the service is in the svn of the Synapse

http://svn.apache.org/viewvc/webservices/synapse/trunk/java/modules/samples/

or else you can get the synapse release binary extract that and go to samples/axis2Server/src/SimpleStockQuoteService and just type ant. This will generate the service archive and put that in to the samples/axis2Server/repository/services folder. You can get it from there.

If you want to do the test on the same environment take a look at the samples guide of synapse

http://ws.apache.org/synapse/Synapse_Samples.html

If you need to any thing clarified please let me know.

Thanks,
Ruwan

[Ruwan Linton]

Sorry, forgot to put the download link for the release,

http://ws.apache.org/synapse/download/1.0/download.cgi

Thanks,
Ruwan

[Thomas Termin]

Bruce,

The problem seems to be in the ProviderProcessor:

You can test this with the following scenario for example:

httpclient -> consumer -> lwcontainer1 -> provider -> NETWORK -> consumer -> lwcontainer2(with sleep of maybe 10sec)

The httpclient has to send more requests in a loop then you have max connections per host.

What happens is all threads (higher then max connections per host) are on lock to get a connection. If the first answer comes back there are no more threads to bring the answer back. And then you have the hang/deadlock. The http connection is closed after the done is initiated for the exchange. If the answer is not processed the done state will never initiated and the connection will not be closed and all other threads wait forever.

To solve this there should be something like if the threads are don't get a connection, they should go back to the pool and the request should be queued. If there is a connection available the next request from the queue can be processed.

Any thoughts on this?

[Guillaume Nodet]

This is a good idea. The only problem is to make sure that if a connection is freed, a job will be automatically dequeued. So this is really a list and we need to manage it ourselves.

Another option is to remove the limitation on the max number of connections and use the thread pool to control this. As there will be no more connections than the number of threads at any time. Well, I guess for the newer endpoints this is wrong because they are asynchronous, but ...

Btw, I think the total number of connections should be set very high in any case, as this is more a system limitation.
And we need to move the thread pool configuration on a per endpoint basis instead of per component imho.

So what if we put both limits to a very high number in the default configuration ? Is there any drawbacks?

[Thomas Termin]

So if I understand you correctly then you will limit the number of connections with the number of threads. I will try this out. But you have to remove the opportunity for the user to change the values for max connections and max connections per host. Because If you don't do that the user can again configure a deadlock scenario, right?

[Guillaume Nodet]

I think that if you allow more connections that the number of threads available, you can not have any deadlock.
I would not remove the way to impose a maximum of connections per host if the default settings are changed in a way that fix the problem (just setting them to Integer.MAX should work). If the user need to have a max number of connections, he could do so, knowing that the number of threads should be lower (or unbounded, in which case all threads would wait for a free connection, but this would not be very interesting)