[CXF-8597] CXF JAXRS client not closing HTTP connections

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.4.3, 3.4.4
• Fix Version/s: 3.5.0, 3.4.5
• Component/s: JAX-RS

Environment

CXF 3.4.3/3.4.4

OpenJDK 11.11

• Estimated Complexity: Unknown
• Assignee: Andriy Redko
• Reporter: Lars Ködderitzsch
• Created: 15/Sep/21 06:17
• Updated: 22/Dec/21 20:41
• Resolved: 28/Sep/21 17:36

Description

After upgrading from cxf-3.3.4 to 3.4.3 we noticed large numbers of HTTP connections remaining in CLOSE_WAIT state, originating from JAXRS proxy clients - unfortunately in production first.
The open sockets lead to a total blockage of the system as file handles eventually are exhausted and no further addtional connections can be opened. At this point we see tens of thousands connections in CLOSE_WAIT.

During our investigation we spotted a substantial refactoring in https://github.com/apache/cxf/blob/75fb6bb56d82f72771a9ee6ecab5d36168303f51/rt/frontend/jaxrs/src/main/java/org/apache/cxf/jaxrs/impl/ResponseImpl.java#L424 which seems to cause the issue (parameter "closeAfterRead" being always false).

Please refer to the attached reproducer project which demonstrates the issue - the underlying HttpUrlConnection never being closed in cxf-3.4.3/3.4.4. By changing the build.gradle.kts of the reproducer to use cxf-3.3.4 the included test case is working. 

 

If possible please advise a workaround. Our initial attempts to get "behind" ResponseImpl#doReadEntity via the client interceptor chain were unsuccessful.

For the time being production operators are forcibly removing CLOSE_WAIT connections by OS means ("ss --tcp state CLOSE-WAIT '( dport = <api-gateway-port> )' --kill")

 

 

Issue Links

• causes: CXF-8601 [Regression] jaxrs.ee.rs.core.response
• links to: GitHub Pull Request #851