[HTTPCORE-754] StrictConnPool unable to acquire lock under multi threaded application causing DeadlineTimeoutException exception being thrown

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 5.2, 5.2.2
• Fix Version/s: 5.2.3, 5.3-alpha1
• Component/s: HttpCore

Language

JAVA Java java

• Assignee: Oleg Kalnichevski
• Reporter: Sreenivas S
• Created: 29/Jun/23 16:57
• Updated: 14/Mar/24 08:22
• Resolved: 06/Jul/23 08:00

Description

Under load when two different threads try to get a connection one thread will be able to get the connection lease and another thread will fail with misleading DeadlineTimeoutException exception.

Exception
[20/Jun/2023:16:08:03-133] [ERROR] - Deadline: [phone]T07:12:55.807+0000, 9223370349577492674 MILLISECONDS overdue
org.apache.hc.core5.util.DeadlineTimeoutException: Deadline: [phone]T07:12:55.807+0000, 9223370349577492674 MILLISECONDS overdue
at org.apache.hc.core5.util.DeadlineTimeoutException.from(DeadlineTimeoutException.java:49) ~[httpcore5-5.2.2.jar:5.2.2]
at org.apache.hc.core5.pool.StrictConnPool.lease(StrictConnPool.java:217) ~[httpcore5-5.2.2.jar:5.2.2]
at org.apache.hc.client5.http.impl.nio.PoolingAsyncClientConnectionManager$3.<init>(PoolingAsyncClientConnectionManager.java:271) ~[httpclient5-5.2.1.jar:5.2.1]
at org.apache.hc.client5.http.impl.nio.PoolingAsyncClientConnectionManager.lease(PoolingAsyncClientConnectionManager.java:266) ~[httpclient5-5.2.1.jar:5.2.1]
at org.apache.hc.client5.http.impl.async.InternalHttpAsyncExecRuntime.acquireEndpoint(InternalHttpAsyncExecRuntime.java:105) ~[httpclient5-5.2.1.jar:5.2.1]
at org.apache.hc.client5.http.impl.async.AsyncConnectExec.execute(AsyncConnectExec.java:141) ~[httpclient5-5.2.1.jar:5.2.1]
at org.apache.hc.client5.http.impl.async.AsyncExecChainElement.execute(AsyncExecChainElement.java:54) ~[httpclient5-5.2.1.jar:5.2.1]
at org.apache.hc.client5.http.impl.async.AsyncProtocolExec.internalExecute(AsyncProtocolExec.java:207) ~[httpclient5-5.2.1.jar:5.2.1]
at org.apache.hc.client5.http.impl.async.AsyncProtocolExec.execute(AsyncProtocolExec.java:172) ~[httpclient5-5.2.1.jar:5.2.1]
at org.apache.hc.client5.http.impl.async.AsyncExecChainElement.execute(AsyncExecChainElement.java:54) ~[httpclient5-5.2.1.jar:5.2.1]
at org.apache.hc.client5.http.impl.async.AsyncHttpRequestRetryExec.internalExecute(AsyncHttpRequestRetryExec.java:97) ~[httpclient5-5.2.1.jar:5.2.1]
at org.apache.hc.client5.http.impl.async.AsyncHttpRequestRetryExec.execute(AsyncHttpRequestRetryExec.java:184) ~[httpclient5-5.2.1.jar:5.2.1]
at org.apache.hc.client5.http.impl.async.AsyncExecChainElement.execute(AsyncExecChainElement.java:54) ~[httpclient5-5.2.1.jar:5.2.1]
at org.apache.hc.client5.http.impl.async.InternalAbstractHttpAsyncClient.executeImmediate(InternalAbstractHttpAsyncClient.java:347) ~[httpclient5-5.2.1.jar:5.2.1]
at org.apache.hc.client5.http.impl.async.InternalAbstractHttpAsyncClient.lambda$doExecute$0(InternalAbstractHttpAsyncClient.java:205) ~[httpclient5-5.2.1.jar:5.2.1]
at org.apache.hc.core5.http.nio.support.BasicRequestProducer.sendRequest(BasicRequestProducer.java:93) ~[httpcore5-5.2.2.jar:5.2.2]
at org.apache.hc.client5.http.impl.async.InternalAbstractHttpAsyncClient.doExecute(InternalAbstractHttpAsyncClient.java:178) ~[httpclient5-5.2.1.jar:5.2.1]
at org.apache.hc.client5.http.impl.async.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:97) ~[httpclient5-5.2.1.jar:5.2.1]
 

DeadlineTimeoutException error is misleading in this case and actually HTTPCore it is unable to acquire lock (From the exception provided earlier Line 217 of StrictConnPool.java). Timeout of 0 means don't wait as per java docs of ReentrantLock. So in my case it looks like already one thread acquired lock and another thread now trying to acquire lock and as TimeOut value is 0, that thread is returning back and causing that exception to be thrown. As per HTTPClient java docs a timeout of 0 to setConnectionRequestTimeout means infinite timeout.

 

Use below code to generate misleading error

         Timeout requestTimeout = Timeout.ZERO_MILLISECONDS;
         final Deadline deadline = Deadline.calculate(requestTimeout);
         System.out.println(DeadlineTimeoutException.from(deadline));