[CXF-7122] Infinite loop due to AsyncHTTPConduit read timeout with exhausted connection pool

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 3.1.9, 3.2.0
• Component/s: Transports
• Estimated Complexity: Unknown
• Assignee: Freeman Yue Fang
• Reporter: William Montaz
• Created: 02/Nov/16 15:34
• Updated: 28/Jan/18 14:21
• Resolved: 10/Nov/16 02:04

Description

Using AsyncHTTPConduit, when the underlying connection pool gets exhausted, requests waiting for a connection will lead to an infinite loop if they reach receive timeout.

The problem occured on all versions of CXF above 3.0.5 (we did not tested other ones).

Let's imagine a backend that's broken and leads to timeout for all requests.

When handling requests, the cxf worker thread will eventually go in wait state (AsyncHTTPConduit:618), with a timeout that matches the HTTPClientPolicy.setReceiveTimeout() value, waiting for the NIO stack to complete and call notifyAll via responseCallback (AsyncHTTPConduit:455).
The timeout on the wait is the big problem :
With our broken backend, the connection pool is exhausted waiting for other requests to timeout. When a new request is made by cxf against this backend, after timeout time this will happen :

• on the one side the reactor threads will get a connection from the pool and try to write to the output stream. Waiting in the pool is not considered as receive timeout.
• on the other side the cxf worker thread will wake up (because of the timedout wait), and shutdown SharedOutputBuffer and SharedInputBuffer (AsyncHTTPClient:624)
• reactor threads will go to infinite loop because they will try to produceContent from a shutdown buffer (SharedOutputBuffer:120)

From there, application recovery is compromised.

To fix that, timeout should be handled only via the client callback (AsyncHTTPConduit:463).

Issue Links

• is related to: CXF-7623 AsynchHTTPConduit setupConnection - timeout settings
• links to: GitHub Pull Request #190