[CXF-5663] IllegalStateException using AsyncHTTPConduit

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.10
• Fix Version/s: 3.1.10
• Component/s: Transports

Environment

SunOS 5.10 Generic_147440-25 sun4v sparc SUNW,SPARC-Enterprise-T5120, 64bit

• Estimated Complexity: Unknown
• Assignee: Freeman Yue Fang
• Reporter: Przemysław Ołtarzewski
• Created: 01/Apr/14 16:01
• Updated: 31/Mar/17 19:38
• Resolved: 23/Mar/17 17:09

Description

We are using CXF with Async HTTP Transport to call web services of another system. Every once in a while an attempt to send request ends in IllegalStateException - Buffer already closed for writing (logs provided below).

2014.04.01 10:47:22.601 org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduitFactory$3 DEBUG [HLAPI-47] DefaultAsyncRequestDirector:208 [exchange: 5663] start execution
2014.04.01 10:47:22.601 org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduitFactory$3 DEBUG [HLAPI-47] DefaultAsyncRequestDirector:625 [exchange: 5663] Request connection for {}->http://10.250.32.140:8082
2014.04.01 10:47:22.602 org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduitFactory$2 DEBUG [HLAPI-47] PoolingClientAsyncConnectionManager:172 Connection request: [route: {}->http://10.250.32.140:8082][total kept alive: 2; route allocated: 2 of 1000; total allocated: 2 of 5000]
2014.04.01 10:47:22.603 org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduitFactory$2 DEBUG [HLAPI-47] PoolingClientAsyncConnectionManager$InternalPoolEntryCallback:297 Connection leased: [id: 1746][route: {}->http://10.250.32.140:8082][total kept alive: 1; route allocated: 2 of 1000; total allocated: 2 of 5000]
2014.04.01 10:47:22.604 org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduitFactory$3 DEBUG [HLAPI-47] DefaultAsyncRequestDirector:558 [exchange: 5663] Connection allocated: [id:1746][route:{}->http://10.250.32.140:8082][state:null]
2014.04.01 10:47:22.625 org.apache.http.impl.nio.reactor.IOSessionImpl DEBUG [I/O dispatcher 18] LoggingIOSession$LoggingByteChannel:189 http-outgoing-1746 10.252.152.177:65127<->10.250.32.140:8082[ACTIVE][r:r]: -1 bytes read
2014.04.01 10:47:22.625 org.apache.http.impl.nio.reactor.IOSessionImpl DEBUG [HLAPI-47] LoggingIOSession:167 http-outgoing-1746 10.252.152.177:65127<->10.250.32.140:8082[ACTIVE][r:r]: Set attribute http.nio.exchange-handler
2014.04.01 10:47:22.626 org.apache.http.impl.nio.reactor.IOSessionImpl DEBUG [I/O dispatcher 18] LoggingIOSession:118 http-outgoing-1746 10.252.152.177:65127<->10.250.32.140:8082[ACTIVE][r:r]: Close
2014.04.01 10:47:22.626 org.apache.http.impl.nio.reactor.IOSessionImpl DEBUG [HLAPI-47] LoggingIOSession:105 http-outgoing-1746 10.252.152.177:65127<->10.250.32.140:8082[CLOSED][]: Event set [w]
2014.04.01 10:47:22.628 org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduitFactory$3 DEBUG [HLAPI-47] DefaultAsyncRequestDirector:185 [exchange: 5663] aborting connection [id:1746][route:{}->http://10.250.32.140:8082][state:null]
2014.04.01 10:47:22.629 org.apache.http.impl.nio.reactor.IOSessionImpl DEBUG [HLAPI-47] LoggingIOSession:133 http-outgoing-1746 [CLOSED][]: Shutdown
2014.04.01 10:47:22.629 org.apache.http.impl.nio.reactor.IOSessionImpl DEBUG [HLAPI-47] LoggingIOSession:133 http-outgoing-1746 [CLOSED][]: Shutdown
2014.04.01 10:47:22.630 org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduitFactory$2 DEBUG [HLAPI-47] PoolingClientAsyncConnectionManager:236 Connection released: [id: 1746][route: {}->http://10.250.32.140:8082][total kept alive: 1; route allocated: 1 of 1000; total allocated: 1 of 5000]
2014.04.01 10:47:22.642 org.apache.cxf.phase.PhaseInterceptorChain WARN  [HLAPI-47] LogUtils:452 Interceptor for {http://implementation.web.api.icc.services.osp.in.alcatel.com}SubscriberLineManagerPortTypeService#{http://implementation.web.api.icc.services.osp.in.alcatel.com}getAll has thrown exception, unwinding now
java.lang.IllegalStateException: IllegalStateException invoking http://10.250.32.140:8082/HighLevelApiWS-2.4/services/SubscriberLineManager: Buffer already closed for writing
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1339)
        at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1328)
        at org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduit$AsyncWrappedOutputStream.close(AsyncHTTPConduit.java:381)
        at org.apache.cxf.io.CacheAndWriteOutputStream.postClose(CacheAndWriteOutputStream.java:50)
        at org.apache.cxf.io.CachedOutputStream.close(CachedOutputStream.java:223)
        at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
        at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:628)
        at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
        at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:272)
        at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:565)
        at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:474)
        at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:377)
        at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:330)
        at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:96)
        at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:135)
        at $Proxy89.getAll(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at pl.infovide.inconfig.bep.backend.hlapi.HlapiCommunicator.send(HlapiCommunicator.java:236)
        at pl.infovide.inconfig.bep.backend.hlapi.HlapiBackend.send(HlapiBackend.java:26)
        at pl.infovide.inconfig.bep.backend.unified.UnifiedBackendImpl.forwardRequest(UnifiedBackendImpl.java:194)
        at pl.infovide.inconfig.bep.backend.unified.UnifiedBackendImpl.access$000(UnifiedBackendImpl.java:29)
        at pl.infovide.inconfig.bep.backend.unified.UnifiedBackendImpl$SenderTask.run(UnifiedBackendImpl.java:276)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.IllegalStateException: Buffer already closed for writing
        at org.apache.cxf.transport.http.asyncclient.SharedOutputBuffer.write(SharedOutputBuffer.java:231)
        at org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduit$AsyncWrappedOutputStream$1.write(AsyncHTTPConduit.java:397)
        at org.apache.cxf.io.AbstractWrappedOutputStream.write(AbstractWrappedOutputStream.java:51)
        at org.apache.cxf.io.AbstractThresholdOutputStream.write(AbstractThresholdOutputStream.java:69)
        at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1296)
        ... 25 common frames omitted

We are unable to provide a test-case since the problem is indeterministic. Roughly 10-20 out of ~14000 requests a day end this way.

We will probably apply a workaround by catching the exception and retrying the request, however CXF's behavior in this case doesn't seem to be valid.