HttpComponents HttpAsyncClient
  1. HttpComponents HttpAsyncClient
  2. HTTPASYNC-3

Async SessionPool does not correctly handle expired I/O sessions

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 4.0-alpha1
    • Fix Version/s: 4.0-alpha2
    • Labels:
      None

      Description

      here is an example code to use the HTTP AsyncClient and seeing an exception. Please also let me know how to deal with this issue,

      /*

      • To change this template, choose Tools | Templates
      • and open the template in the editor.
        */
        package httpanalysis.apache;

      import java.util.concurrent.CountDownLatch;
      import java.util.concurrent.Future;
      import java.util.concurrent.TimeUnit;
      import java.util.logging.Level;
      import java.util.logging.Logger;
      import org.apache.http.HttpHost;

      import org.apache.http.HttpResponse;
      import org.apache.http.client.methods.HttpGet;
      import org.apache.http.impl.nio.client.DefaultHttpAsyncClient;
      import org.apache.http.impl.nio.conn.PoolingClientConnectionManager;
      import org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor;
      import org.apache.http.nio.client.HttpAsyncClient;
      import org.apache.http.nio.conn.scheme.Scheme;
      import org.apache.http.nio.conn.scheme.SchemeRegistry;
      import org.apache.http.nio.reactor.ConnectingIOReactor;
      import org.apache.http.nio.reactor.IOReactorException;
      import org.apache.http.params.BasicHttpParams;
      import org.apache.http.params.CoreConnectionPNames;

      /**
      *

      • @author lokesh
        */
        public class HttpAnalysis {

      PoolingClientConnectionManager poolManager;
      HttpAsyncClient httpclient = null;
      private PoolingClientConnectionManager sessionManager;

      /**

      • @param args the command line arguments
        */
        public static void main(String[] args) { HttpAnalysis analysis = new HttpAnalysis(); analysis.process(); }

      public HttpAnalysis() {
      try

      { BasicHttpParams basicHttpParams = new BasicHttpParams(); basicHttpParams.setParameter(CoreConnectionPNames.CONNECTION_TIMEOUT, 1); basicHttpParams.setParameter(CoreConnectionPNames.SO_TIMEOUT, 2); basicHttpParams.setParameter(CoreConnectionPNames.SOCKET_BUFFER_SIZE, 2 * 1024); basicHttpParams.setParameter(CoreConnectionPNames.STALE_CONNECTION_CHECK, false); basicHttpParams.setParameter(CoreConnectionPNames.SO_REUSEADDR, false); ConnectingIOReactor ioReactor = new DefaultConnectingIOReactor(2, basicHttpParams); SchemeRegistry schemeRegistry = new SchemeRegistry(); schemeRegistry.register(new Scheme("http", 80, null)); this.sessionManager = new PoolingClientConnectionManager(ioReactor, schemeRegistry, 5, TimeUnit.MINUTES); sessionManager.setTotalMax(50); sessionManager.setDefaultMaxPerHost(25); this.httpclient = new DefaultHttpAsyncClient(ioReactor, sessionManager,basicHttpParams); }

      catch (IOReactorException ex)

      { Logger.getLogger(HttpAnalysis.class.getName()).log(Level.SEVERE, null, ex); }
      }

      private void process() {
      try {
      int numRequests = 10000;
      httpclient.start();
      long startTime = System.currentTimeMillis();
      final CountDownLatch countDownLatch = new CountDownLatch(numRequests);
      for (int i = 0; i < numRequests; i++) {
      HttpGet request = new HttpGet("http://hc.apache.org/");
      Future<HttpResponse> future = httpclient.execute(request, new HttpCallback(this, countDownLatch));
      if(future == null){ countDownLatch.countDown(); }
      System.out.println("Request number = " + i);
      //sessionManager.closeExpiredConnections();
      }
      countDownLatch.await();
      System.out.println((System.currentTimeMillis() - startTime));
      System.exit(1);
      } catch (Exception ex) { Logger.getLogger(HttpAnalysis.class.getName()).log(Level.SEVERE, null, ex); }

      }
      }

        Activity

        Hide
        Denis Dzenskevich added a comment -

        Yes, now works fine in application as well as in tests.

        Show
        Denis Dzenskevich added a comment - Yes, now works fine in application as well as in tests.
        Hide
        Oleg Kalnichevski added a comment -

        @Denis

        Right you are. I found the problem and fixed it in SVN trunk. Please re-test your application against the latest SVN snapshot and confirm the fix.

        Oleg

        Show
        Oleg Kalnichevski added a comment - @Denis Right you are. I found the problem and fixed it in SVN trunk. Please re-test your application against the latest SVN snapshot and confirm the fix. Oleg
        Hide
        Denis Dzenskevich added a comment -

        Does NOT work.

        I also found the deterministic test, maybe it can help with debugging. I use Jetty here to simulate slow response and socket timeout. The errors are similar in both this test and the one originally reported.

        import org.apache.http.HttpResponse;
        import org.apache.http.client.methods.HttpGet;
        import org.apache.http.impl.nio.client.DefaultHttpAsyncClient;
        import org.apache.http.impl.nio.conn.PoolingClientConnectionManager;
        import org.apache.http.nio.concurrent.FutureCallback;
        import org.mortbay.jetty.Server;
        import org.mortbay.jetty.nio.SelectChannelConnector;
        import org.mortbay.jetty.servlet.Context;
        import org.mortbay.jetty.servlet.ServletHolder;
        import javax.servlet.ServletException;
        import javax.servlet.http.HttpServlet;
        import javax.servlet.http.HttpServletRequest;
        import javax.servlet.http.HttpServletResponse;
        import java.io.IOException;

        public class Test2 {

        public static void main(String[] args) throws Exception {
        final Server server = new Server(8080);
        server.addConnector(new SelectChannelConnector());
        Context root = new Context(server, "/", Context.NO_SESSIONS);
        ServletHolder servletHolder = new ServletHolder(new SimpleServlet());
        root.addServlet(servletHolder, "/");
        Thread serverThread = new Thread(new Runnable() {
        public void run() {
        try

        { server.start(); }

        catch (Exception e)

        { e.printStackTrace(); }

        }
        });
        serverThread.start();
        while (!server.isStarted())
        Thread.sleep(0);
        DefaultHttpAsyncClient client = new DefaultHttpAsyncClient();
        client.getParams().setParameter("http.socket.timeout", 1000);
        ((PoolingClientConnectionManager)client.getConnectionManager()).setDefaultMaxPerHost(1);
        ((PoolingClientConnectionManager)client.getConnectionManager()).setTotalMax(1);
        client.start();
        client.execute(new HttpGet("http://localhost:8080/1"), new MyFutureCallback(1));
        client.execute(new HttpGet("http://localhost:8080/2"), new MyFutureCallback(2))
        .get();
        System.exit(0);
        }

        public static class SimpleServlet extends HttpServlet {
        @Override protected void service(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
        try

        { Thread.sleep(100000); }

        catch (InterruptedException ignore) {
        }
        resp.setContentType("text/plain");
        resp.setCharacterEncoding("utf-8");
        resp.setStatus(HttpServletResponse.SC_OK);
        resp.getWriter().println("Requested: " + req.getRequestURI());
        }
        }

        private static class MyFutureCallback implements FutureCallback<HttpResponse> {

        private final int i;

        public MyFutureCallback(int i)

        { this.i = i; }

        public void completed(HttpResponse result)

        { System.err.println("completed: " + i + ": " + result.getStatusLine().getStatusCode()); }

        public void failed(Exception ex)

        { System.err.println("failed: " + i + ": " + ex.getMessage()); }

        public void cancelled()

        { System.err.println("cancelled: " + i); }

        }
        }

        Show
        Denis Dzenskevich added a comment - Does NOT work. I also found the deterministic test, maybe it can help with debugging. I use Jetty here to simulate slow response and socket timeout. The errors are similar in both this test and the one originally reported. import org.apache.http.HttpResponse; import org.apache.http.client.methods.HttpGet; import org.apache.http.impl.nio.client.DefaultHttpAsyncClient; import org.apache.http.impl.nio.conn.PoolingClientConnectionManager; import org.apache.http.nio.concurrent.FutureCallback; import org.mortbay.jetty.Server; import org.mortbay.jetty.nio.SelectChannelConnector; import org.mortbay.jetty.servlet.Context; import org.mortbay.jetty.servlet.ServletHolder; import javax.servlet.ServletException; import javax.servlet.http.HttpServlet; import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletResponse; import java.io.IOException; public class Test2 { public static void main(String[] args) throws Exception { final Server server = new Server(8080); server.addConnector(new SelectChannelConnector()); Context root = new Context(server, "/", Context.NO_SESSIONS); ServletHolder servletHolder = new ServletHolder(new SimpleServlet()); root.addServlet(servletHolder, "/"); Thread serverThread = new Thread(new Runnable() { public void run() { try { server.start(); } catch (Exception e) { e.printStackTrace(); } } }); serverThread.start(); while (!server.isStarted()) Thread.sleep(0); DefaultHttpAsyncClient client = new DefaultHttpAsyncClient(); client.getParams().setParameter("http.socket.timeout", 1000); ((PoolingClientConnectionManager)client.getConnectionManager()).setDefaultMaxPerHost(1); ((PoolingClientConnectionManager)client.getConnectionManager()).setTotalMax(1); client.start(); client.execute(new HttpGet("http://localhost:8080/1"), new MyFutureCallback(1)); client.execute(new HttpGet("http://localhost:8080/2"), new MyFutureCallback(2)) .get(); System.exit(0); } public static class SimpleServlet extends HttpServlet { @Override protected void service(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException { try { Thread.sleep(100000); } catch (InterruptedException ignore) { } resp.setContentType("text/plain"); resp.setCharacterEncoding("utf-8"); resp.setStatus(HttpServletResponse.SC_OK); resp.getWriter().println("Requested: " + req.getRequestURI()); } } private static class MyFutureCallback implements FutureCallback<HttpResponse> { private final int i; public MyFutureCallback(int i) { this.i = i; } public void completed(HttpResponse result) { System.err.println("completed: " + i + ": " + result.getStatusLine().getStatusCode()); } public void failed(Exception ex) { System.err.println("failed: " + i + ": " + ex.getMessage()); } public void cancelled() { System.err.println("cancelled: " + i); } } }
        Hide
        Oleg Kalnichevski added a comment -

        Fixed in SVN trunk.

        @Lokesh & Denis
        Could you please re-test your applications against the latest SVN snapshot and confirm the fix?

        Oleg

        Show
        Oleg Kalnichevski added a comment - Fixed in SVN trunk. @Lokesh & Denis Could you please re-test your applications against the latest SVN snapshot and confirm the fix? Oleg
        Hide
        Oleg Kalnichevski added a comment -

        It turned out the root cause of the problem was a bug in the expired I/O sessions handling logic. The test application posted by the reporter uses very aggressive timeout settings causing many connections to time out, which eventually leads to an inconsistent state in the SessionPool internal data structures.

        Oleg

        Show
        Oleg Kalnichevski added a comment - It turned out the root cause of the problem was a bug in the expired I/O sessions handling logic. The test application posted by the reporter uses very aggressive timeout settings causing many connections to time out, which eventually leads to an inconsistent state in the SessionPool internal data structures. Oleg
        Hide
        Denis Dzenskevich added a comment -

        I used a
        private class HttpCallback implements FutureCallback<HttpResponse> {
        public void completed(HttpResponse result)

        { System.out.println("completed!"); countDownLatch.countDown(); }

        public void failed(Exception ex)

        { System.out.println("failed: " + ex.getMessage()); countDownLatch.countDown(); }

        public void cancelled()

        { System.out.println("cancelled!"); countDownLatch.countDown(); }

        which is the simplest that fits the code.
        With it, I was able to reproduce the issue. Then, after applying my patch the errors were gone. So, I assumed that it was the same issue that I saw.
        Let's see if it helps with the problem reported. I will try to isolate my findings in a test case meanwhile.

        Show
        Denis Dzenskevich added a comment - I used a private class HttpCallback implements FutureCallback<HttpResponse> { public void completed(HttpResponse result) { System.out.println("completed!"); countDownLatch.countDown(); } public void failed(Exception ex) { System.out.println("failed: " + ex.getMessage()); countDownLatch.countDown(); } public void cancelled() { System.out.println("cancelled!"); countDownLatch.countDown(); } which is the simplest that fits the code. With it, I was able to reproduce the issue. Then, after applying my patch the errors were gone. So, I assumed that it was the same issue that I saw. Let's see if it helps with the problem reported. I will try to isolate my findings in a test case meanwhile.
        Hide
        Oleg Kalnichevski added a comment -

        Denis

        The I/O reactor used internally by HttpAsyncClient to manage low level I/O sessions is programmed to shut down itself in case of any unexpected (usually runtime) exception. There can be various reasons why this can happen. Most common cause are logical errors in custom request producers / response consumers. I suspect that the issue encountered by the original reporter is likely to be caused by his/her own HttpCallback class that under certain conditions throws a runtime exception.

        I guess in your case we are dealing with a genuine bug in HttpAsyncClient. Could you please open a new JIRA for the issue and attach the stack trace of the exception that caused the I/O reactor to shut down and if possible a test case reproducing the problem?

        Oleg

        Show
        Oleg Kalnichevski added a comment - Denis The I/O reactor used internally by HttpAsyncClient to manage low level I/O sessions is programmed to shut down itself in case of any unexpected (usually runtime) exception. There can be various reasons why this can happen. Most common cause are logical errors in custom request producers / response consumers. I suspect that the issue encountered by the original reporter is likely to be caused by his/her own HttpCallback class that under certain conditions throws a runtime exception. I guess in your case we are dealing with a genuine bug in HttpAsyncClient. Could you please open a new JIRA for the issue and attach the stack trace of the exception that caused the I/O reactor to shut down and if possible a test case reproducing the problem? Oleg
        Hide
        Denis Dzenskevich added a comment -

        I recently came into the same issue
        The following seems to work (line numbers can differ slightly because of IDE reformat imports):

        — httpasyncclient/src/main/java/org/apache/http/impl/nio/pool/SessionPool.java (revision 1096308)
        +++ httpasyncclient/src/main/java/org/apache/http/impl/nio/pool/SessionPool.java (working copy)
        @@ -199,7 +193,8 @@
        E entry = this.availableSessions.remove();
        entryShutdown(entry);
        RouteSpecificPool<T, E> pool = getPool(entry.getRoute());

        • pool.freeEntry(entry, false);
          + if (this.leasedSessions.contains(entry))
          + throw new IllegalStateException("shouldn't happen");
          }
          }

        From debugger, availableSessions and leasedSessions does not overlap, so no need to remove it from leasedSessions which is what freeEntry does. Trying to freeEnty then throws an exception and must be leaving connection in bad state which causes subsequent errors.

        Show
        Denis Dzenskevich added a comment - I recently came into the same issue The following seems to work (line numbers can differ slightly because of IDE reformat imports): — httpasyncclient/src/main/java/org/apache/http/impl/nio/pool/SessionPool.java (revision 1096308) +++ httpasyncclient/src/main/java/org/apache/http/impl/nio/pool/SessionPool.java (working copy) @@ -199,7 +193,8 @@ E entry = this.availableSessions.remove(); entryShutdown(entry); RouteSpecificPool<T, E> pool = getPool(entry.getRoute()); pool.freeEntry(entry, false); + if (this.leasedSessions.contains(entry)) + throw new IllegalStateException("shouldn't happen"); } } From debugger, availableSessions and leasedSessions does not overlap, so no need to remove it from leasedSessions which is what freeEntry does. Trying to freeEnty then throws an exception and must be leaving connection in bad state which causes subsequent errors.
        Hide
        Oleg Kalnichevski added a comment -

        I cannot reproduce the problem. Code does not compile because HttpCallback class is not defined.

        Oleg

        Show
        Oleg Kalnichevski added a comment - I cannot reproduce the problem. Code does not compile because HttpCallback class is not defined. Oleg

          People

          • Assignee:
            Unassigned
            Reporter:
            Lokesh
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 1m
              1m
              Remaining:
              Remaining Estimate - 1m
              1m
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development