Details
Major Description
Our platform allows the user to switch transports for a particular proxy based on configuration. We currently allow HTTP and JMS as possible options.
During recent performance testing, we've hit a bug which causes JAXRS Clients over JMS to fail with a ConcurrentModificationException:
A test case is attached that reproduces our problem.
Our test executes requests across multiple concurrent threads, switching between JMS and HTTP for both JAXWS and JAXRS Client Proxies. In the test case:
- JAXWS over both HTTP and JMS works fine, both in single and multi-threaded tests.
- JAXRS works fine across single and multi-threaded tests over HTTP with the 'ThreadSafe' flag set on the Client Factory.
- JAXRS over JMS fails in both single-threaded and multi-threaded mode with error (1) below. In multi-threaded, also fails with (2) below.
1) The ConnectionFactoryFeature fails to properly set the JMS Connection Factory on the Conduit and results in an error at send-time because no Connection Factory is set. This is because in JAXRS Clients, the InterceptorProvider is neither a Client or Server, it is a ClientConfiguration, so the JMS ConnectionFactoryFeature does not fire either of it's methods:
public void initialize(Client client, Bus bus)
public void initialize(Server server, Bus bus)
We have worked around this in our code by sub-classing and overriding the method:
public void initialize(InterceptorProvider interceptorProvider, Bus bus)
.. and adding code similar to that in the ConnectionFactory feature. This resolves the missing ConnectionFactory problem and JAXRS-over-JMS works fine in single-threaded mode.
2) Under multi-threaded conditions, the JAXRS Clients fail with the following exception:
javax.ws.rs.client.ResponseProcessingException: Problem with reading the data, class java.lang.String, ContentType: text/plain.
at org.apache.cxf.jaxrs.impl.ResponseImpl.reportMessageHandlerProblem(ResponseImpl.java:438)
at org.apache.cxf.jaxrs.impl.ResponseImpl.doReadEntity(ResponseImpl.java:378)
at org.apache.cxf.jaxrs.client.AbstractClient.readBody(AbstractClient.java:521)
at org.apache.cxf.jaxrs.client.ClientProxyImpl.handleResponse(ClientProxyImpl.java:817)
at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:760)
at org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:228)
at com.sun.proxy.$Proxy24.doSimpleMethod(Unknown Source)
at com.ge.testcase.ThreadingTest$ThreadedRSJMSTest.run(ThreadingTest.java:132)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1429)
at java.util.HashMap$EntryIterator.next(HashMap.java:1463)
at java.util.HashMap$EntryIterator.next(HashMap.java:1461)
at java.util.HashMap.putMapEntries(HashMap.java:511)
at java.util.HashMap.putAll(HashMap.java:784)
at org.apache.cxf.message.MessageImpl$1.putAll(MessageImpl.java:188)
at org.apache.cxf.message.MessageImpl.calcContextCache(MessageImpl.java:213)
at org.apache.cxf.message.MessageImpl.getContextualProperty(MessageImpl.java:174)
at org.apache.cxf.jaxrs.impl.HttpHeadersImpl.getRequestHeaders(HttpHeadersImpl.java:172)
at org.apache.cxf.jaxrs.utils.JAXRSUtils.readFromMessageBodyReader(JAXRSUtils.java:1345)
at org.apache.cxf.jaxrs.impl.ResponseImpl.doReadEntity(ResponseImpl.java:369)
... 7 more
This occurs even though the 'ThreadSafe' property is set via:
rsClientFactory.setThreadSafe(true);
Also, interestingly, this exception occurs even if I put the entire Proxy Client invocation in a synchronized block (which should effectively render the Clients single-threaded).
Finally, one important factor is that this error only happens on the latest 3.0.x release (3.0.11). When I switch my test case to 3.1.8, it does NOT occur. Not sure if this equates to a fix not being back-ported, or a fundamental difference in functionality. Sadly, we're not able to yet move to 3.1.x, because of running inside the Karaf container, and our dependencies on Camel and Spring 3.x.
I've done quite a large amount of digging through the code to try and narrow down the scope of the problem, and the results above is as far as I could get, unfortunately. My guess/assumption is that there are some JMS-specific objects that are being added to the Request/Response context which are being added/removed in a non-thread safe way. Purely a guess, though.
Thanks. Please let me know if you need any other information.