Details
-
Bug
-
Status: Closed
-
Critical
-
Resolution: Fixed
-
3.1.5, 3.0.8
-
None
-
Unknown
Description
We have a spring boot application that is doing SOAP over JMS with a tibco backend.
Whenever the JMS connection is reset our application fails to receive the answer for the SOA request until the application server is restarted.
From the logs we see that re-establishing the jms connection is working fine:
2016-02-05 12:59:03.458 WARN 5662 — [TIBCO EMS TCPLink Reader (Server-608470)] o.s.j.c.CachingConnectionFactory : Encountered a JMSException - resetting the underlying JMS Connection
javax.jms.JMSException: Connection has been terminated
at com.tibco.tibjms.Tibjmsx.buildException(Tibjmsx.java:509)
at com.tibco.tibjms.TibjmsConnection._invokeOnExceptionCallback(TibjmsConnection.java:2025)
at com.tibco.tibjms.TibjmsConnection._onDisconnected(TibjmsConnection.java:2394)
at com.tibco.tibjms.TibjmsConnection$ServerLinkEventHandler.onEventDisconnected(TibjmsConnection.java:349)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:330)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)
2016-02-05 12:59:04.701 INFO 5662 — [ajp-bio-18009-exec-1] o.s.j.c.CachingConnectionFactory: Established shared JMS Connection: QueueConnection[ClientId=null Connected=tcp://ems2-k:12004, URL=tcp://ems2-k:12004]
After the connection has been reset when our application does the next SOAP call sending the request works fine, but when CXF waits for the reply the following exception occurs:
javax.xml.ws.soap.SOAPFaultException: Timeout receiving message with correlationId b02ad4683421442db439b54797fcc7350000000000000003
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:161)
Debugging shows that the problem seems to be located in public Destination getReplyDestination(Session session) in org.apache.cxf.transport.jms.JMSConfiguration. There a Destination object for receiving the answer is created once and cached forever. There is no way this cached destination will ever be dropped and recreated. This destination object (implemented by a temporary queue in our case) contains a reference to the jms connection. So after a reset of the connection it still contains the old, invalid jms connection object. This is why after a connection reset it is impossible to receive any more replies.
Using a debugger, setting a breakpoint in this method and manually setting replyDestinationDest to null forces this method to recreate the temporary queue. After that receiving replies is working again after a jms connection reset. This is of course not possible for our live environment, there all servers have to be restarted after a jms connection reset.
This behaviour was introduced in CXF3. Before upgrading we used CXF 2 where the temporary response queue seems not to be cached but created and deleted for every request. With CXF 2 our application worked fine when the jms connection was reset.
Steps for reproducing this error (sorry, no example project yet):
- Have a SOAP server
- Have a SOAP client using Spring so that CachingConnectionFactory will do a proper connection reset
- Have a SOAP over JMS broker
- Make one successful client request so that the replyDestinationDest gets created and cached.
- Force a JMS connection reset (e.g. drop connection from the broker admin console)
- Try another client request. The client will reconnect but re-use the old replyDestinationDest and never be able to receive the reply again.
Attachments
Issue Links
- Is contained by
-
-
- Closed
-