[CXF-6940] Bus Reference nullified pre-maturely in ClientImpl causing Exceptions while processing Responses

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Resolved
• Affects Version/s: 3.1.4
• Fix Version/s: Invalid
• Component/s: JAX-WS Runtime

Environment

IBM JDK 1.7, Apache CXF installed as resource Adapter on IBM WebSphere 8.5.5.3 on RHEL 6.0

• Estimated Complexity: Guru
• Flags: Important
• Assignee: Daniel Kulp
• Reporter: Dhawalpatel
• Created: 15/Jun/16 00:21
• Updated: 21/Oct/16 18:45
• Resolved: 21/Jun/16 15:53

Description

Hi Daniel,

Test Scenario:
Install CXF as Resource Adapter in J2EE Container and use below code to call a JAX-WS API.

/* Lookup Resource Adapter of JCA and get COnnection*/

CXFConnectionFactory connectionFactory = getCXFConnectionFactory(portClass);
CXFConnectionSpec connectionSpec = createCXFConnectionSpec(portClass, serviceName, portName, webServiceDefinition);
/* Get the Port from JCA Connection pool*/

connection = connectionFactory.getConnection(connectionSpec);
port = connection.getService(portClass);
/* Close the JCA Connection*/
if(connection != null)
connection.close();

/WebService call/
port.getAccount();

Description:
When a JCA Connection pool eviction happens due to Max Connections reached state, the existing on-going outgoing JAX-WS Calls errors out with below exception:

Message received on a Client that has been closed or destroyed.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:86)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:58)
at java.lang.reflect.Constructor.newInstance(Constructor.java:542)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1376)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1365)
at org.apache.cxf.io.CacheAndWriteOutputStream.postClose(CacheAndWriteOutputStream.java:56)
at org.apache.cxf.io.CachedOutputStream.close(CachedOutputStream.java:215)
at org.apache.cxf.io.AbstractWrappedOutputStream.close(AbstractWrappedOutputStream.java:77)
at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:651)
at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:520)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:429)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:330)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:283)
at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:96)
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:139)
at com.sun.proxy.$Proxy106.inquireWirelinePaymentPlan(Unknown Source)
at sun.reflect.GeneratedMethodAccessor360.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)
at java.lang.reflect.Method.invoke(Method.java:620)
at AppAdapter.executeWebService(AppAdapter.java:702)

Caused by:
java.lang.IllegalStateException: Message received on a Client that has been closed or destroyed.
at org.apache.cxf.endpoint.ClientImpl.onMessage(ClientImpl.java:707)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1669)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1550)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1347)
... 26 more

On further investigation, it seems you changed the code in ClientImpl.java as per https://issues.apache.org/jira/browse/CXF-3765 and change log
https://fisheye6.atlassian.com/changelog/cxf?cs=e0ef66391d2e7e62e63e96214eca95e292271892

by adding the additional cleanup code in ClientImpl.java the below lines

bus = null;
conduitSelector = null;
outFaultObserver = null;
outboundChainCache = null;
inboundChainCache = null;

currentRequestContext = null;
requestContext.clear();
requestContext = null;
responseContext.clear();
responseContext = null;
executor = null;

The above lines added by you for cleanup has broken the behavior of using the JCA connection for getting port and closing it immediately and then using the port to call the webservice. This leads to above exception when JCA Pool maxes out causing an eviction on JCA Pool by below thread stack causing the ClientImpl's destroy to be called and ultimately the bus obj ref getting set to null by your code and ultimately this exception.
The behavior in CXF 2.3 was perfectly fine with no such problems. This was introduced by above mentioned code fix.

org.apache.cxf.endpoint.ClientImpl.destroy(ClientImpl.java:152)
org.apache.cxf.jca.outbound.ManagedConnectionImpl.destroy(ManagedConnectionImpl.java:121)
com.ibm.ejs.j2c.MCWrapper.destroy(MCWrapper.java:2086)
com.ibm.ejs.j2c.FreePool.cleanupAndDestroyMCWrapper(FreePool.java:775)
com.ibm.ejs.j2c.PoolManager.claimVictim(PoolManager.java:5190)
com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3657)
com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3091)
com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java:1548)
com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java:1031)
org.apache.cxf.jca.outbound.ConnectionFactoryImpl.getConnection(ConnectionFactoryImpl.java:53)
com.sbc.cc.clarify.api.util.cxf.XCxfConnectorWebServiceInitializer.getCxfPortImpl(XCxfConnectorWebServiceInitializer.java:217)

Please fix this behavior since its not in the alignment to the behavior mentioned in JCA Spec and as per the documentation provided in CXF as well.
Thanks

Issue Links

• relates to: CXF-2398 JMS Connections are not closed when JCA recycles CXF proxy
• links to: GitHub Pull Request #143