[HTTPCORE-587] Connection must be created by connection manager issue

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 4.4.11
• Component/s: HttpCore, HttpCore NIO
• Assignee: Unassigned
• Reporter: Pranav Arora
• Created: 31/Jul/19 07:41
• Updated: 15/Aug/19 16:25
• Resolved: 31/Jul/19 08:34

Description

This is the same issue that is discussed in HTTPCORE-370 and HTTPCORE-574. I am facing the following exception:

 

 

[ERROR] [] 2019-07-29 09:33:10.639 [pool-2-thread-1] InternalHttpAsyncClient - I/O reactor terminated abnormally
org.apache.http.nio.reactor.IOReactorException: I/O dispatch worker terminated abnormally
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:356)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]
Caused by: java.lang.IllegalStateException: Connection must be created by connection manager
at org.apache.http.impl.nio.client.InternalIODispatch.createConnection(InternalIODispatch.java:56)
at org.apache.http.impl.nio.client.InternalIODispatch.createConnection(InternalIODispatch.java:39)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.connected(AbstractIODispatch.java:70)
at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionCreated(BaseIOReactor.java:248)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processNewChannels(AbstractIOReactor.java:427)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:287)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
... 1 more

 

This occurs randomly during production and I have not been able to reproduce it locally. What I instead tried was pass a custom exception handler. 

 

 

public class CustomExceptionHandler implements IOReactorExceptionHandler {
 @Override
 public boolean handle(IOException ex) {
  log.error("I/O Reactor IO exception", ex);
  return false;
 }

 @Override
 public boolean handle(RuntimeException ex) {
  log.error("I/O Reactor Runtime Exception", ex);
  if (ex instanceof IllegalStateException) {
   throw new CancelledKeyException();
  }
  return false;
 }
}

 

 

It helped for the above exception as it did not lead to immediate shutdown of I/O Reactor but now I am facing a different issue.

 

 

[ERROR] [] 2019-07-30 18:22:37.684 [pool-2-thread-1] InternalHttpAsyncClient - I/O reactor terminated abnormally
java.lang.NullPointerException: null
at <packageNameTruncated>.DefaultConnectingIOReactorTemp.processEvent(DefaultConnectingIOReactorTemp.java:158)
at <packageNameTruncated>.DefaultConnectingIOReactorTemp.processEvents(DefaultConnectingIOReactorTemp.java:136)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
at java.lang.Thread.run(Thread.java:748)

 

 

DefaultConnectingIOReactorTemp is exactly the same as DefaultConnectingIOReactor with the added code that was intended to fix HTTPCORE-370. The corresponding line in DefaultConnectingIOReactor is 167 (final SessionRequestImpl sessionRequest = requestHandle.getSessionRequest()). The added code is this: https://github.com/ok2c/httpcore/commit/aa812282f26fdd1975233a892c5405fa0da781b4#diff-d577e717cb1e97f3a3c0adbc8d563062

 

Issue Links

• duplicates: HTTPCORE-574 Race condition if connection request succeeds and times out at the same time