Uploaded image for project: 'HttpComponents HttpAsyncClient'
  1. HttpComponents HttpAsyncClient
  2. HTTPASYNC-152

"Connection must be created by connection manager" raised when requests are reused after being aborted

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 4.1.4
    • Fix Version/s: 4.1.5
    • Labels:
      None

      Description

      We are using http async client in the official Java REST client for Elasticsearch. We recently introduced the ability to cancel requests. While testing such new feature, I encountered a test failure that can be reproduced when the same request instance is reused (although reset is called after each run) after being cancelled. The following exception is raised, which calls the reactor to shut down unexpectedly, which fails all subsequent requests:

      Aug 15, 2019 2:04:52 PM org.apache.http.impl.nio.client.InternalHttpAsyncClient run
      SEVERE: I/O reactor terminated abnormally
      org.apache.http.nio.reactor.IOReactorException: I/O dispatch worker terminated abnormally
      	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:359)
      	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
      	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
      	at java.base/java.lang.Thread.run(Thread.java:844)
      Caused by: java.lang.IllegalStateException: Connection must be created by connection manager
      	at org.apache.http.impl.nio.client.InternalIODispatch.createConnection(InternalIODispatch.java:56)
      	at org.apache.http.impl.nio.client.InternalIODispatch.createConnection(InternalIODispatch.java:39)
      	at org.apache.http.impl.nio.reactor.AbstractIODispatch.connected(AbstractIODispatch.java:70)
      	at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionCreated(BaseIOReactor.java:248)
      	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processNewChannels(AbstractIOReactor.java:427)
      	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:287)
      	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
      	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
      	... 1 more
      
      Exception in thread "main" java.util.concurrent.ExecutionException: java.lang.IllegalStateException: I/O reactor has been shut down
      	at org.apache.http.concurrent.BasicFuture.getResult(BasicFuture.java:71)
      	at org.apache.http.concurrent.BasicFuture.get(BasicFuture.java:84)
      	at org.apache.http.impl.nio.client.FutureWrapper.get(FutureWrapper.java:70)
      	at ClientTest.main(ClientTest.java:31)
      Caused by: java.lang.IllegalStateException: I/O reactor has been shut down
      	at org.apache.http.util.Asserts.check(Asserts.java:34)
      	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.connect(DefaultConnectingIOReactor.java:228)
      	at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(AbstractNIOConnPool.java:481)
      	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:280)
      	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.java:295)
      	at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:377)
      	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:129)
      	at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141)
      	at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:75)
      	at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:85)
      	at ClientTest.main(ClientTest.java:28)
      

      The following snippet reproduces it most of the times:

      import com.sun.net.httpserver.HttpServer;
      import org.apache.http.HttpHost;
      import org.apache.http.HttpResponse;
      import org.apache.http.client.methods.HttpGet;
      import org.apache.http.impl.nio.client.CloseableHttpAsyncClient;
      import org.apache.http.impl.nio.client.HttpAsyncClientBuilder;
      
      import java.net.InetAddress;
      import java.net.InetSocketAddress;
      import java.util.concurrent.CancellationException;
      import java.util.concurrent.Future;
      
      public class ClientTest {
      
          public static void main(String[] args) throws Exception {
              HttpServer httpServer = HttpServer.create(new InetSocketAddress(InetAddress.getLoopbackAddress(), 0), 0);
              httpServer.start();
              httpServer.createContext("/test", exchange -> {
                  exchange.sendResponseHeaders(200, 0);
                  exchange.close();
              });
              HttpHost httpHost = new HttpHost(httpServer.getAddress().getHostString(), httpServer.getAddress().getPort());
              try (CloseableHttpAsyncClient client = HttpAsyncClientBuilder.create().build()) {
                  client.start();
                  HttpGet httpGet = new HttpGet("/test");
                  for (int i = 0; i < 10000; i++) {
                      httpGet.reset();
                      Future<HttpResponse> future = client.execute(httpHost, httpGet, null);
                      httpGet.abort();
                      try {
                          future.get();
                          assert false;
                      } catch(CancellationException e) {
                          //expected
                      }
                  }
              } finally {
                  httpServer.stop(0);
              }
          }
      }
      
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                olegk Oleg Kalnichevski
                Reporter:
                lucacavanna Luca Cavanna
              • Votes:
                1 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 40m
                  40m