Ivy
  1. Ivy
  2. IVY-1105

Excessive hits to missing URLs uses up all ports on Windows

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.0-RC1
    • Fix Version/s: 2.2.0-RC1
    • Component/s: Core
    • Labels:
      None
    • Environment:

      windows xp sp3, running Ivy repository through http for remote resolving

      Description

      Modified description to be clearer:
      Ivy is extremely aggressive towards repositories . This can result in
      failures to resolve, even against a healthy repository and with no network latency.

      The symptom:

      [ivy:resolve] 01-07-2008 13:16:24
      org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
      [ivy:resolve] INFO: I/O exception (java.net.BindException) caught when
      processing request: Address already in use: connect.

      This is especially a problem with large repositories (lots of revisions) and resolving against latest.integration as this will fetch ivy.xml md5 and sha1 files for every revision.

      As I've included in a comment, the problem appears to be Ivy's attempts to list URLs that don't actually exist. Ivy builds up URLs using the repository location and appending organizations and module names. We have ivy chains that contain 5 links or more. So for each successful attempt to list an URL, there are several attempts to list non-existent URLs. This is wasteful in terms of time, and can lead to using up all of the available Windows ports on the client machine.

        Issue Links

          Activity

          Hide
          Maarten Coene added a comment -

          ok, so I'll mark this issue as fixed.
          Please reopen if you still have problems.

          Thanks for the help in solving this issue!

          Maarten

          Show
          Maarten Coene added a comment - ok, so I'll mark this issue as fixed. Please reopen if you still have problems. Thanks for the help in solving this issue! Maarten
          Hide
          Ernest Pasour added a comment -

          That looks like it helps quite a bit! I now see perhaps one port being left around in TIME_WAIT state per resolve, which is much better than lots of ports being left around per resolve. I'll try to confirm tomorrow with a more taxing example.

          Show
          Ernest Pasour added a comment - That looks like it helps quite a bit! I now see perhaps one port being left around in TIME_WAIT state per resolve, which is much better than lots of ports being left around per resolve. I'll try to confirm tomorrow with a more taxing example.
          Hide
          Maarten Coene added a comment -

          I've committed a potential fix in SVN trunk.
          Could you please give it a try to see if it fixes the problem?
          I couldn't test it because I'm on windows Vista right now which doens't seem to have the problem (butI could reproduce the problem earlier this day on a windows XP machine)

          Show
          Maarten Coene added a comment - I've committed a potential fix in SVN trunk. Could you please give it a try to see if it fixes the problem? I couldn't test it because I'm on windows Vista right now which doens't seem to have the problem (butI could reproduce the problem earlier this day on a windows XP machine)
          Hide
          Ernest Pasour added a comment - - edited

          Attached my hackaround. This suffers from general caching shortcomings, but has the side benefit of reducing unnecessary server connections. Minimally tested. I'm just including it here as a possibility if the underlying problem can't be fixed without a Java sdk fix.

          Show
          Ernest Pasour added a comment - - edited Attached my hackaround. This suffers from general caching shortcomings, but has the side benefit of reducing unnecessary server connections. Minimally tested. I'm just including it here as a possibility if the underlying problem can't be fixed without a Java sdk fix.
          Hide
          Ernest Pasour added a comment -

          hackaround that prevents the problem by capturing bad URLs and preventing them from being accessed again

          Show
          Ernest Pasour added a comment - hackaround that prevents the problem by capturing bad URLs and preventing them from being accessed again
          Hide
          Ernest Pasour added a comment -

          ...but it does not appear that our Ant setup includes any version of http-client.jar. I'll have to check on that tomorrow.

          Show
          Ernest Pasour added a comment - ...but it does not appear that our Ant setup includes any version of http-client.jar. I'll have to check on that tomorrow.
          Hide
          Ernest Pasour added a comment -

          It appears to be using BasicURLHandler. While I was debugging, I set breakpoints in HttpClientHandler and did not see them hit.

          We have also reproduced this problem running from Ant (in the process of both doing builds), but it's a little more difficult to tell what's going on there.

          Show
          Ernest Pasour added a comment - It appears to be using BasicURLHandler. While I was debugging, I set breakpoints in HttpClientHandler and did not see them hit. We have also reproduced this problem running from Ant (in the process of both doing builds), but it's a little more difficult to tell what's going on there.
          Hide
          Maarten Coene added a comment -

          Could you tell what URLHandler is used? For instance, what class does URLHandlerRegistry.getDefault() returns?
          As far as I can tell, commons-httpclient will only be used if you use the Ant tasks or use the Ivy CLI.

          Show
          Maarten Coene added a comment - Could you tell what URLHandler is used? For instance, what class does URLHandlerRegistry.getDefault() returns? As far as I can tell, commons-httpclient will only be used if you use the Ant tasks or use the Ivy CLI.
          Hide
          Ernest Pasour added a comment -

          I already had httpclient 3.0 in my classpath. I updated to 3.1 and it didn't seem to make any difference. I think httpclient is on the classpath properly. I am running the Ivy source code (2.0) and using Ivy API methods directly if that makes a difference.

          Show
          Ernest Pasour added a comment - I already had httpclient 3.0 in my classpath. I updated to 3.1 and it didn't seem to make any difference. I think httpclient is on the classpath properly. I am running the Ivy source code (2.0) and using Ivy API methods directly if that makes a difference.
          Hide
          Maarten Coene added a comment -

          ok, I see the problem and I was able to fix the problem illustrated by your main-method, now I'll have to apply the same approach it to Ivy's codebase.
          Can you confirm that this problem does not occur when you add commons-httpclient-3.x to Ivy's classpath?

          Show
          Maarten Coene added a comment - ok, I see the problem and I was able to fix the problem illustrated by your main-method, now I'll have to apply the same approach it to Ivy's codebase. Can you confirm that this problem does not occur when you add commons-httpclient-3.x to Ivy's classpath?
          Hide
          Jim Adams added a comment -

          I think some of the initial wording is a little odd. The idea that this is a DOS attack against the server is not right. In actuality it seems like a client performing a DOS attack on itself. In any case I think that portion of the wording is not right. The client is unusable until the socket times out.

          Show
          Jim Adams added a comment - I think some of the initial wording is a little odd. The idea that this is a DOS attack against the server is not right. In actuality it seems like a client performing a DOS attack on itself. In any case I think that portion of the wording is not right. The client is unusable until the socket times out.
          Hide
          Ernest Pasour added a comment -

          It appears that the problem we are experiencing right now is because of all the attempts to read non-existent directories using the ApacheURLLister class. When the connection attempts (and fails) to get the input stream, it is apparently enough to create a connection, but the connection is not cleaned up properly, even using the disconnect call. The code below reproduces the problem (on windows) if you put the proper data in the url definition. You will get the BindException after about 15 seconds. If you do a "netstat" from the command line, you will see lots of connections in the TIME_WAIT state.

          I think Nascif's idea above makes sense to reduce the number of connections being made. I think it would also make sense to reduce the number of invalid URLs used. I hacked together some code to keep a hash of the shortest invalid urls and then not attempt to make connections when subsequent urls show up with an invalid prefix. This seems to work but I haven't really validated it yet. This idea could also be used in conjunction with Nascif's idea.

          import java.io.BufferedReader;
          import java.io.IOException;
          import java.io.InputStreamReader;
          import java.net.BindException;
          import java.net.HttpURLConnection;
          import java.net.URL;
          import java.net.URLConnection;
          public class Test2 {
          public static void main(String[] args) throws IOException {
          for (int i=0;;i++)
          {
          URL url=new URL("http", "<your server>", 80, "<bad path on server>"); //shows problem
          // URL url=new URL("http", "<your server>", 80, "<existing path on server>"); //works fine; does not leak connections
          URLConnection conn=url.openConnection();
          String htmlText = "";
          try

          { BufferedReader r = new BufferedReader(new InputStreamReader(conn.getInputStream())); }

          catch (BindException e)

          { e.printStackTrace(); }

          catch (IOException e)
          {}
          finally
          {
          if (conn instanceof HttpURLConnection)

          { ((HttpURLConnection)conn).disconnect(); }

          }
          }
          }
          }

          Show
          Ernest Pasour added a comment - It appears that the problem we are experiencing right now is because of all the attempts to read non-existent directories using the ApacheURLLister class. When the connection attempts (and fails) to get the input stream, it is apparently enough to create a connection, but the connection is not cleaned up properly, even using the disconnect call. The code below reproduces the problem (on windows) if you put the proper data in the url definition. You will get the BindException after about 15 seconds. If you do a "netstat" from the command line, you will see lots of connections in the TIME_WAIT state. I think Nascif's idea above makes sense to reduce the number of connections being made. I think it would also make sense to reduce the number of invalid URLs used. I hacked together some code to keep a hash of the shortest invalid urls and then not attempt to make connections when subsequent urls show up with an invalid prefix. This seems to work but I haven't really validated it yet. This idea could also be used in conjunction with Nascif's idea. import java.io.BufferedReader; import java.io.IOException; import java.io.InputStreamReader; import java.net.BindException; import java.net.HttpURLConnection; import java.net.URL; import java.net.URLConnection; public class Test2 { public static void main(String[] args) throws IOException { for (int i=0;;i++) { URL url=new URL("http", "<your server>", 80, "<bad path on server>"); //shows problem // URL url=new URL("http", "<your server>", 80, "<existing path on server>"); //works fine; does not leak connections URLConnection conn=url.openConnection(); String htmlText = ""; try { BufferedReader r = new BufferedReader(new InputStreamReader(conn.getInputStream())); } catch (BindException e) { e.printStackTrace(); } catch (IOException e) {} finally { if (conn instanceof HttpURLConnection) { ((HttpURLConnection)conn).disconnect(); } } } } }
          Hide
          Ernest Pasour added a comment -

          From Nascif:
          I believe we are seeing the same issue on Ivy 2.0 final, will post more details soon.

          I have to agree that the current design is very much a DOS attack on the server. Our scenario is very much as described above as well, large number of modules, with large number of revisions, and resolving for latest. We can easily get to thousands of connections created and destroyed in a resolve, and we are bumping into the Windows limitations described in http://support.microsoft.com/default.aspx?scid=kb;EN-US;196271. For continuous integration builds, this is a major issue.

          I wonder if a different approach could be taken for the url resolver. Instead of releasing the connection on every download() and upload(), why not take advantage of HTTP 1.1 ability to keep connections open? You would go from thousands of connections per resolve to 1 - and it would be constant, instead of on the order of modules x versions x resolvers as it is today.

          Since there is already a concept of locking in Ivy, it means there is a concept of transaction borders. You could use that to implement to setup and closing of the connection.
          http://java.sun.com/j2se/1.5.0/docs/guide/net/http-keepalive.html

          Show
          Ernest Pasour added a comment - From Nascif: I believe we are seeing the same issue on Ivy 2.0 final, will post more details soon. I have to agree that the current design is very much a DOS attack on the server. Our scenario is very much as described above as well, large number of modules, with large number of revisions, and resolving for latest. We can easily get to thousands of connections created and destroyed in a resolve, and we are bumping into the Windows limitations described in http://support.microsoft.com/default.aspx?scid=kb;EN-US;196271 . For continuous integration builds, this is a major issue. I wonder if a different approach could be taken for the url resolver. Instead of releasing the connection on every download() and upload(), why not take advantage of HTTP 1.1 ability to keep connections open? You would go from thousands of connections per resolve to 1 - and it would be constant, instead of on the order of modules x versions x resolvers as it is today. Since there is already a concept of locking in Ivy, it means there is a concept of transaction borders. You could use that to implement to setup and closing of the connection. http://java.sun.com/j2se/1.5.0/docs/guide/net/http-keepalive.html
          Hide
          Ernest Pasour added a comment -

          I've cloned https://issues.apache.org/jira/browse/IVY-854 because we appear to have the same issue, but I don't know how to change the status of it back to "Open". I'm copying the comments Nascif and I made on it into this comment for easier reference.

          Show
          Ernest Pasour added a comment - I've cloned https://issues.apache.org/jira/browse/IVY-854 because we appear to have the same issue, but I don't know how to change the status of it back to "Open". I'm copying the comments Nascif and I made on it into this comment for easier reference.

            People

            • Assignee:
              Maarten Coene
              Reporter:
              Ernest Pasour
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development