Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 4.0-alpha1
    • 4.0-alpha2
    • None

    Description

      here is an example code to use the HTTP AsyncClient and seeing an exception. Please also let me know how to deal with this issue,

      /*

      • To change this template, choose Tools | Templates
      • and open the template in the editor.
        */
        package httpanalysis.apache;

      import java.util.concurrent.CountDownLatch;
      import java.util.concurrent.Future;
      import java.util.concurrent.TimeUnit;
      import java.util.logging.Level;
      import java.util.logging.Logger;
      import org.apache.http.HttpHost;

      import org.apache.http.HttpResponse;
      import org.apache.http.client.methods.HttpGet;
      import org.apache.http.impl.nio.client.DefaultHttpAsyncClient;
      import org.apache.http.impl.nio.conn.PoolingClientConnectionManager;
      import org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor;
      import org.apache.http.nio.client.HttpAsyncClient;
      import org.apache.http.nio.conn.scheme.Scheme;
      import org.apache.http.nio.conn.scheme.SchemeRegistry;
      import org.apache.http.nio.reactor.ConnectingIOReactor;
      import org.apache.http.nio.reactor.IOReactorException;
      import org.apache.http.params.BasicHttpParams;
      import org.apache.http.params.CoreConnectionPNames;

      /**
      *

      • @author lokesh
        */
        public class HttpAnalysis {

      PoolingClientConnectionManager poolManager;
      HttpAsyncClient httpclient = null;
      private PoolingClientConnectionManager sessionManager;

      /**

      • @param args the command line arguments
        */
        public static void main(String[] args) { HttpAnalysis analysis = new HttpAnalysis(); analysis.process(); }

      public HttpAnalysis() {
      try

      { BasicHttpParams basicHttpParams = new BasicHttpParams(); basicHttpParams.setParameter(CoreConnectionPNames.CONNECTION_TIMEOUT, 1); basicHttpParams.setParameter(CoreConnectionPNames.SO_TIMEOUT, 2); basicHttpParams.setParameter(CoreConnectionPNames.SOCKET_BUFFER_SIZE, 2 * 1024); basicHttpParams.setParameter(CoreConnectionPNames.STALE_CONNECTION_CHECK, false); basicHttpParams.setParameter(CoreConnectionPNames.SO_REUSEADDR, false); ConnectingIOReactor ioReactor = new DefaultConnectingIOReactor(2, basicHttpParams); SchemeRegistry schemeRegistry = new SchemeRegistry(); schemeRegistry.register(new Scheme("http", 80, null)); this.sessionManager = new PoolingClientConnectionManager(ioReactor, schemeRegistry, 5, TimeUnit.MINUTES); sessionManager.setTotalMax(50); sessionManager.setDefaultMaxPerHost(25); this.httpclient = new DefaultHttpAsyncClient(ioReactor, sessionManager,basicHttpParams); }

      catch (IOReactorException ex)

      { Logger.getLogger(HttpAnalysis.class.getName()).log(Level.SEVERE, null, ex); }
      }

      private void process() {
      try {
      int numRequests = 10000;
      httpclient.start();
      long startTime = System.currentTimeMillis();
      final CountDownLatch countDownLatch = new CountDownLatch(numRequests);
      for (int i = 0; i < numRequests; i++) {
      HttpGet request = new HttpGet("http://hc.apache.org/");
      Future<HttpResponse> future = httpclient.execute(request, new HttpCallback(this, countDownLatch));
      if(future == null){ countDownLatch.countDown(); }
      System.out.println("Request number = " + i);
      //sessionManager.closeExpiredConnections();
      }
      countDownLatch.await();
      System.out.println((System.currentTimeMillis() - startTime));
      System.exit(1);
      } catch (Exception ex) { Logger.getLogger(HttpAnalysis.class.getName()).log(Level.SEVERE, null, ex); }

      }
      }

      Attachments

        Activity

          I cannot reproduce the problem. Code does not compile because HttpCallback class is not defined.

          Oleg

          olegk Oleg Kalnichevski added a comment - I cannot reproduce the problem. Code does not compile because HttpCallback class is not defined. Oleg

          I recently came into the same issue
          The following seems to work (line numbers can differ slightly because of IDE reformat imports):

          — httpasyncclient/src/main/java/org/apache/http/impl/nio/pool/SessionPool.java (revision 1096308)
          +++ httpasyncclient/src/main/java/org/apache/http/impl/nio/pool/SessionPool.java (working copy)
          @@ -199,7 +193,8 @@
          E entry = this.availableSessions.remove();
          entryShutdown(entry);
          RouteSpecificPool<T, E> pool = getPool(entry.getRoute());

          • pool.freeEntry(entry, false);
            + if (this.leasedSessions.contains(entry))
            + throw new IllegalStateException("shouldn't happen");
            }
            }

          From debugger, availableSessions and leasedSessions does not overlap, so no need to remove it from leasedSessions which is what freeEntry does. Trying to freeEnty then throws an exception and must be leaving connection in bad state which causes subsequent errors.

          _loki_ Denis Dzenskevich added a comment - I recently came into the same issue The following seems to work (line numbers can differ slightly because of IDE reformat imports): — httpasyncclient/src/main/java/org/apache/http/impl/nio/pool/SessionPool.java (revision 1096308) +++ httpasyncclient/src/main/java/org/apache/http/impl/nio/pool/SessionPool.java (working copy) @@ -199,7 +193,8 @@ E entry = this.availableSessions.remove(); entryShutdown(entry); RouteSpecificPool<T, E> pool = getPool(entry.getRoute()); pool.freeEntry(entry, false); + if (this.leasedSessions.contains(entry)) + throw new IllegalStateException("shouldn't happen"); } } From debugger, availableSessions and leasedSessions does not overlap, so no need to remove it from leasedSessions which is what freeEntry does. Trying to freeEnty then throws an exception and must be leaving connection in bad state which causes subsequent errors.

          Denis

          The I/O reactor used internally by HttpAsyncClient to manage low level I/O sessions is programmed to shut down itself in case of any unexpected (usually runtime) exception. There can be various reasons why this can happen. Most common cause are logical errors in custom request producers / response consumers. I suspect that the issue encountered by the original reporter is likely to be caused by his/her own HttpCallback class that under certain conditions throws a runtime exception.

          I guess in your case we are dealing with a genuine bug in HttpAsyncClient. Could you please open a new JIRA for the issue and attach the stack trace of the exception that caused the I/O reactor to shut down and if possible a test case reproducing the problem?

          Oleg

          olegk Oleg Kalnichevski added a comment - Denis The I/O reactor used internally by HttpAsyncClient to manage low level I/O sessions is programmed to shut down itself in case of any unexpected (usually runtime) exception. There can be various reasons why this can happen. Most common cause are logical errors in custom request producers / response consumers. I suspect that the issue encountered by the original reporter is likely to be caused by his/her own HttpCallback class that under certain conditions throws a runtime exception. I guess in your case we are dealing with a genuine bug in HttpAsyncClient. Could you please open a new JIRA for the issue and attach the stack trace of the exception that caused the I/O reactor to shut down and if possible a test case reproducing the problem? Oleg

          I used a
          private class HttpCallback implements FutureCallback<HttpResponse> {
          public void completed(HttpResponse result)

          { System.out.println("completed!"); countDownLatch.countDown(); }

          public void failed(Exception ex)

          { System.out.println("failed: " + ex.getMessage()); countDownLatch.countDown(); }

          public void cancelled()

          { System.out.println("cancelled!"); countDownLatch.countDown(); }

          which is the simplest that fits the code.
          With it, I was able to reproduce the issue. Then, after applying my patch the errors were gone. So, I assumed that it was the same issue that I saw.
          Let's see if it helps with the problem reported. I will try to isolate my findings in a test case meanwhile.

          _loki_ Denis Dzenskevich added a comment - I used a private class HttpCallback implements FutureCallback<HttpResponse> { public void completed(HttpResponse result) { System.out.println("completed!"); countDownLatch.countDown(); } public void failed(Exception ex) { System.out.println("failed: " + ex.getMessage()); countDownLatch.countDown(); } public void cancelled() { System.out.println("cancelled!"); countDownLatch.countDown(); } which is the simplest that fits the code. With it, I was able to reproduce the issue. Then, after applying my patch the errors were gone. So, I assumed that it was the same issue that I saw. Let's see if it helps with the problem reported. I will try to isolate my findings in a test case meanwhile.

          It turned out the root cause of the problem was a bug in the expired I/O sessions handling logic. The test application posted by the reporter uses very aggressive timeout settings causing many connections to time out, which eventually leads to an inconsistent state in the SessionPool internal data structures.

          Oleg

          olegk Oleg Kalnichevski added a comment - It turned out the root cause of the problem was a bug in the expired I/O sessions handling logic. The test application posted by the reporter uses very aggressive timeout settings causing many connections to time out, which eventually leads to an inconsistent state in the SessionPool internal data structures. Oleg

          Fixed in SVN trunk.

          @Lokesh & Denis
          Could you please re-test your applications against the latest SVN snapshot and confirm the fix?

          Oleg

          olegk Oleg Kalnichevski added a comment - Fixed in SVN trunk. @Lokesh & Denis Could you please re-test your applications against the latest SVN snapshot and confirm the fix? Oleg

          Does NOT work.

          I also found the deterministic test, maybe it can help with debugging. I use Jetty here to simulate slow response and socket timeout. The errors are similar in both this test and the one originally reported.

          import org.apache.http.HttpResponse;
          import org.apache.http.client.methods.HttpGet;
          import org.apache.http.impl.nio.client.DefaultHttpAsyncClient;
          import org.apache.http.impl.nio.conn.PoolingClientConnectionManager;
          import org.apache.http.nio.concurrent.FutureCallback;
          import org.mortbay.jetty.Server;
          import org.mortbay.jetty.nio.SelectChannelConnector;
          import org.mortbay.jetty.servlet.Context;
          import org.mortbay.jetty.servlet.ServletHolder;
          import javax.servlet.ServletException;
          import javax.servlet.http.HttpServlet;
          import javax.servlet.http.HttpServletRequest;
          import javax.servlet.http.HttpServletResponse;
          import java.io.IOException;

          public class Test2 {

          public static void main(String[] args) throws Exception {
          final Server server = new Server(8080);
          server.addConnector(new SelectChannelConnector());
          Context root = new Context(server, "/", Context.NO_SESSIONS);
          ServletHolder servletHolder = new ServletHolder(new SimpleServlet());
          root.addServlet(servletHolder, "/");
          Thread serverThread = new Thread(new Runnable() {
          public void run() {
          try

          { server.start(); }

          catch (Exception e)

          { e.printStackTrace(); }

          }
          });
          serverThread.start();
          while (!server.isStarted())
          Thread.sleep(0);
          DefaultHttpAsyncClient client = new DefaultHttpAsyncClient();
          client.getParams().setParameter("http.socket.timeout", 1000);
          ((PoolingClientConnectionManager)client.getConnectionManager()).setDefaultMaxPerHost(1);
          ((PoolingClientConnectionManager)client.getConnectionManager()).setTotalMax(1);
          client.start();
          client.execute(new HttpGet("http://localhost:8080/1"), new MyFutureCallback(1));
          client.execute(new HttpGet("http://localhost:8080/2"), new MyFutureCallback(2))
          .get();
          System.exit(0);
          }

          public static class SimpleServlet extends HttpServlet {
          @Override protected void service(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
          try

          { Thread.sleep(100000); }

          catch (InterruptedException ignore) {
          }
          resp.setContentType("text/plain");
          resp.setCharacterEncoding("utf-8");
          resp.setStatus(HttpServletResponse.SC_OK);
          resp.getWriter().println("Requested: " + req.getRequestURI());
          }
          }

          private static class MyFutureCallback implements FutureCallback<HttpResponse> {

          private final int i;

          public MyFutureCallback(int i)

          { this.i = i; }

          public void completed(HttpResponse result)

          { System.err.println("completed: " + i + ": " + result.getStatusLine().getStatusCode()); }

          public void failed(Exception ex)

          { System.err.println("failed: " + i + ": " + ex.getMessage()); }

          public void cancelled()

          { System.err.println("cancelled: " + i); }

          }
          }

          _loki_ Denis Dzenskevich added a comment - Does NOT work. I also found the deterministic test, maybe it can help with debugging. I use Jetty here to simulate slow response and socket timeout. The errors are similar in both this test and the one originally reported. import org.apache.http.HttpResponse; import org.apache.http.client.methods.HttpGet; import org.apache.http.impl.nio.client.DefaultHttpAsyncClient; import org.apache.http.impl.nio.conn.PoolingClientConnectionManager; import org.apache.http.nio.concurrent.FutureCallback; import org.mortbay.jetty.Server; import org.mortbay.jetty.nio.SelectChannelConnector; import org.mortbay.jetty.servlet.Context; import org.mortbay.jetty.servlet.ServletHolder; import javax.servlet.ServletException; import javax.servlet.http.HttpServlet; import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletResponse; import java.io.IOException; public class Test2 { public static void main(String[] args) throws Exception { final Server server = new Server(8080); server.addConnector(new SelectChannelConnector()); Context root = new Context(server, "/", Context.NO_SESSIONS); ServletHolder servletHolder = new ServletHolder(new SimpleServlet()); root.addServlet(servletHolder, "/"); Thread serverThread = new Thread(new Runnable() { public void run() { try { server.start(); } catch (Exception e) { e.printStackTrace(); } } }); serverThread.start(); while (!server.isStarted()) Thread.sleep(0); DefaultHttpAsyncClient client = new DefaultHttpAsyncClient(); client.getParams().setParameter("http.socket.timeout", 1000); ((PoolingClientConnectionManager)client.getConnectionManager()).setDefaultMaxPerHost(1); ((PoolingClientConnectionManager)client.getConnectionManager()).setTotalMax(1); client.start(); client.execute(new HttpGet("http://localhost:8080/1"), new MyFutureCallback(1)); client.execute(new HttpGet("http://localhost:8080/2"), new MyFutureCallback(2)) .get(); System.exit(0); } public static class SimpleServlet extends HttpServlet { @Override protected void service(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException { try { Thread.sleep(100000); } catch (InterruptedException ignore) { } resp.setContentType("text/plain"); resp.setCharacterEncoding("utf-8"); resp.setStatus(HttpServletResponse.SC_OK); resp.getWriter().println("Requested: " + req.getRequestURI()); } } private static class MyFutureCallback implements FutureCallback<HttpResponse> { private final int i; public MyFutureCallback(int i) { this.i = i; } public void completed(HttpResponse result) { System.err.println("completed: " + i + ": " + result.getStatusLine().getStatusCode()); } public void failed(Exception ex) { System.err.println("failed: " + i + ": " + ex.getMessage()); } public void cancelled() { System.err.println("cancelled: " + i); } } }

          @Denis

          Right you are. I found the problem and fixed it in SVN trunk. Please re-test your application against the latest SVN snapshot and confirm the fix.

          Oleg

          olegk Oleg Kalnichevski added a comment - @Denis Right you are. I found the problem and fixed it in SVN trunk. Please re-test your application against the latest SVN snapshot and confirm the fix. Oleg

          Yes, now works fine in application as well as in tests.

          _loki_ Denis Dzenskevich added a comment - Yes, now works fine in application as well as in tests.

          People

            Unassigned Unassigned
            lokeshhctm Lokesh
            Votes:
            0 Vote for this issue
            Watchers:
            Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - 1m
                1m
                Remaining:
                Remaining Estimate - 1m
                1m
                Logged:
                Time Spent - Not Specified
                Not Specified

                Slack

                  Issue deployment