Uploaded image for project: 'CXF'
  1. CXF
  2. CXF-7122

Infinite loop due to AsyncHTTPConduit read timeout with exhausted connection pool

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.1.9, 3.2.0
    • Component/s: Transports
    • Labels:
      None
    • Estimated Complexity:
      Unknown

      Description

      Using AsyncHTTPConduit, when the underlying connection pool gets exhausted, requests waiting for a connection will lead to an infinite loop if they reach receive timeout.

      The problem occured on all versions of CXF above 3.0.5 (we did not tested other ones).

      Let's imagine a backend that's broken and leads to timeout for all requests.

      When handling requests, the cxf worker thread will eventually go in wait state (AsyncHTTPConduit:618), with a timeout that matches the HTTPClientPolicy.setReceiveTimeout() value, waiting for the NIO stack to complete and call notifyAll via responseCallback (AsyncHTTPConduit:455).
      The timeout on the wait is the big problem :
      With our broken backend, the connection pool is exhausted waiting for other requests to timeout. When a new request is made by cxf against this backend, after timeout time this will happen :

      • on the one side the reactor threads will get a connection from the pool and try to write to the output stream. Waiting in the pool is not considered as receive timeout.
      • on the other side the cxf worker thread will wake up (because of the timedout wait), and shutdown SharedOutputBuffer and SharedInputBuffer (AsyncHTTPClient:624)
      • reactor threads will go to infinite loop because they will try to produceContent from a shutdown buffer (SharedOutputBuffer:120)

      From there, application recovery is compromised.

      To fix that, timeout should be handled only via the client callback (AsyncHTTPConduit:463).

        Attachments

        1. AsyncHTTPConduitTest.java
          14 kB
          Freeman Yue Fang

          Issue Links

            Activity

              People

              • Assignee:
                ffang Freeman Yue Fang
                Reporter:
                willymontaz William Montaz
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: