[CXF-7109] ClientCallback may be invoked twice when Async HTTP Transport is used

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.8
• Fix Version/s: 3.1.9, 3.2.0
• Component/s: Core, Transports

Environment

RHEL 6.8

• Estimated Complexity: Unknown
• Assignee: Freeman Yue Fang
• Reporter: Tadayoshi Sato
• Created: 25/Oct/16 09:34
• Updated: 22/Mar/17 17:24
• Resolved: 26/Oct/16 06:10

Description

With Async HTTP Transport (cxf-rt-transports-http-hc) enabled, the ClientCallback object passed for the invocation of the client may be called twice when a failure occurs during the invocation:

Client client = ...
client.invoke(new ClientCallback { ... }, opName, params);

At this moment, this issue is reproduced only on RHEL 6.8, but code analysis shows that it's a general multithreading issue with Async HTTP Transport and in theory it can happen on any platform.

Here is what I've found with the code analysis:

If for some reason the target endpoint is not reachable, this issue can happen when the callback for Apache HC HttpAsyncClient [1] is invoked and thus shuts down SharedOutputBuffer earlier than HTTPConduit tries to write it before closing the conduit itself [2].
[1] https://github.com/apache/cxf/blob/cxf-3.1.8/rt/transports/http-hc/src/main/java/org/apache/cxf/transport/http/asyncclient/AsyncHTTPConduit.java#L464
[2] https://github.com/apache/cxf/blob/cxf-3.1.8/rt/transports/http-hc/src/main/java/org/apache/cxf/transport/http/asyncclient/SharedOutputBuffer.java#L231

When the above issue happens, the interceptor chain's fault observer is eventually invoked twice, once from the response workqueue thread [3] that is spawn by the Apache HC HttpAsyncClient callback and the second time from the requesting thread at the interceptor chain [4]; Thus the client callback is also invoked twice [5]!
[3] https://github.com/apache/cxf/blob/cxf-3.1.8/rt/transports/http/src/main/java/org/apache/cxf/transport/http/HTTPConduit.java#L1177
[4] https://github.com/apache/cxf/blob/cxf-3.1.8/core/src/main/java/org/apache/cxf/phase/PhaseInterceptorChain.java#L366
[5] https://github.com/apache/cxf/blob/cxf-3.1.8/core/src/main/java/org/apache/cxf/interceptor/ClientOutFaultObserver.java#L59

Why it happens so reliably only on RHEL 6.8 is still unclear (probably due to some thread scheduling logic specific to the kernel for RHEL 6.8), but since it's essentially an ordering issue for two distinct threads, in theory it can happen on any platforms. In general, CXF should really enforce that every client callback be invoked once and only once.

Issue Links

• links to: GitHub Pull Request #187