[HTTPCLIENT-2184] Leased Connection Leak after upgrade to 5.1.1

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 5.1.1
• Fix Version/s: 5.1.2
• Component/s: HttpClient (classic)
• Assignee: Carter Kozak
• Reporter: Carter Kozak
• Created: 11/Nov/21 17:09
• Updated: 13/Nov/21 10:59
• Resolved: 12/Nov/21 22:44

Description

Unfortunately I haven't had a chance to build a local reproducer yet, nor have I had a chance to profile a system experiencing this failure – I intend to add additional data to this ticket later on as I debug.

I do have a system deployed where the issue does not reproduce on 5.1, but I can redeploy only changing the httpcomponents-client jar to 5.1.1 (no other transitive dependency changes, etc) and the leak appears. This should rule out bad application code as a root cause, I also have metrics which capture any time a response is not closed, which reports zero leaks in both cases.

I have attached a graph of my connection pool leased-connections metric, the blue line in the middle represents the first version which took httpcomponents-client 5.1.1, where previously leased connections sat right around zero. There's a short gap after the blue line ends where we rolled back to the previous release, followed by a few smaller successful attempts to reproduce the issue resulting in large sets of leased connections.