Infrastructure
  1. Infrastructure
  2. INFRA-5151

Intermittent connectivity issues with Maven ASF site

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Fix Version/s: Initial Clearing
    • Component/s: None
    • Labels:
      None

      Description

      Hi,

      We are observing intermittent connectivity problems with the ASF Maven site.

      An example of a failed request (to 140.211.11.131): http://pastie.org/4522672
      If we wait a bit, the request eventually succeeds (to 192.87.106.229): http://pastie.org/4522680

      It seems that there is some round-robin DNS happing somewhere. However, I am unable to replicate the problem (though responses from 140.211.11.131 seem slower than 192.87.106.229), e.g. with:
      curl --verbose -H "Host: maven.apache.org" http://140.211.11.131/plugins/maven-javadoc-plugin/examples/test-javadocs.html

      I wasn't sure if a) this is a problem on our side, or b) if infra was aware of any issues.

      Regards,
      Elliot

        Activity

        Hide
        Elliot Metsger added a comment -
        Content of: http://pastie.org/pastes/4522672/text

        $ curl --verbose http://maven.apache.org/plugins/maven-javadoc-plugin/examples/test-javadocs.html
        * About to connect() to maven.apache.org port 80 (#0)
        * Trying 140.211.11.131... connected
        * Connected to maven.apache.org (140.211.11.131) port 80 (#0)
        > GET /plugins/maven-javadoc-plugin/examples/test-javadocs.html HTTP/1.1
        > User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5
        > Host: maven.apache.org
        > Accept: */*
        >
        * Recv failure: Connection reset by peer
        * Closing connection #0
        curl: (56) Recv failure: Connection reset by peer
        Show
        Elliot Metsger added a comment - Content of: http://pastie.org/pastes/4522672/text $ curl --verbose http://maven.apache.org/plugins/maven-javadoc-plugin/examples/test-javadocs.html * About to connect() to maven.apache.org port 80 (#0) * Trying 140.211.11.131... connected * Connected to maven.apache.org (140.211.11.131) port 80 (#0) > GET /plugins/maven-javadoc-plugin/examples/test-javadocs.html HTTP/1.1 > User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5 > Host: maven.apache.org > Accept: */* > * Recv failure: Connection reset by peer * Closing connection #0 curl: (56) Recv failure: Connection reset by peer
        Hide
        Elliot Metsger added a comment -
        Content of: http://pastie.org/pastes/4522680/text
        $ curl --verbose http://maven.apache.org/plugins/maven-javadoc-plugin/examples/test-javadocs.html
        * About to connect() to maven.apache.org port 80 (#0)
        * Trying 192.87.106.229... connected
        * Connected to maven.apache.org (192.87.106.229) port 80 (#0)
        > GET /plugins/maven-javadoc-plugin/examples/test-javadocs.html HTTP/1.1
        > User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5
        > Host: maven.apache.org
        > Accept: */*
        >
        < HTTP/1.1 200 OK
        < Date: Thu, 16 Aug 2012 17:54:57 GMT
        < Server: Apache/2.4.1 (Unix) OpenSSL/1.0.0g
        < Last-Modified: Mon, 23 Jan 2012 15:42:54 GMT
        < ETag: "3cda-4b733e2417b80"
        < Accept-Ranges: bytes
        < Content-Length: 15578
        < Vary: Accept-Encoding
        < Content-Type: text/html; charset=utf-8
        <
        <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
        .....
        Show
        Elliot Metsger added a comment - Content of: http://pastie.org/pastes/4522680/text $ curl --verbose http://maven.apache.org/plugins/maven-javadoc-plugin/examples/test-javadocs.html * About to connect() to maven.apache.org port 80 (#0) * Trying 192.87.106.229... connected * Connected to maven.apache.org (192.87.106.229) port 80 (#0) > GET /plugins/maven-javadoc-plugin/examples/test-javadocs.html HTTP/1.1 > User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5 > Host: maven.apache.org > Accept: */* > < HTTP/1.1 200 OK < Date: Thu, 16 Aug 2012 17:54:57 GMT < Server: Apache/2.4.1 (Unix) OpenSSL/1.0.0g < Last-Modified: Mon, 23 Jan 2012 15:42:54 GMT < ETag: "3cda-4b733e2417b80" < Accept-Ranges: bytes < Content-Length: 15578 < Vary: Accept-Encoding < Content-Type: text/html; charset=utf-8 < <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" " http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd "> .....
        Hide
        #asfinfra IRC Bot added a comment -
        <danielsh> www.apache.org (which resolves to the two IPs you mention) had problems.
        Show
        #asfinfra IRC Bot added a comment - <danielsh> www.apache.org (which resolves to the two IPs you mention) had problems.
        Hide
        Elliot Metsger added a comment -
        Ok, thanks for the update. We've been seeing these issues over the last couple of days. The fact that it was an ongoing issue is what prompted me to file the Jira issue. Thanks again for the update.
        Show
        Elliot Metsger added a comment - Ok, thanks for the update. We've been seeing these issues over the last couple of days. The fact that it was an ongoing issue is what prompted me to file the Jira issue. Thanks again for the update.
        Hide
        #asfinfra IRC Bot added a comment -
        <danielsh> Please check http://status.apache.org if you haven't
        Show
        #asfinfra IRC Bot added a comment - <danielsh> Please check http://status.apache.org if you haven't
        Hide
        Elliot Metsger added a comment -
        This still seems to be an issue:

        esm:~/dcs-docs-git$ curl --verbose http://maven.apache.org/doxia/doxia/index.html
        * About to connect() to maven.apache.org port 80 (#0)
        * Trying 140.211.11.131... connected
        * Connected to maven.apache.org (140.211.11.131) port 80 (#0)
        > GET /doxia/doxia/index.html HTTP/1.1
        > User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5
        > Host: maven.apache.org
        > Accept: */*
        >
        * Recv failure: Connection reset by peer
        * Closing connection #0
        curl: (56) Recv failure: Connection reset by peer
        esm:~/dcs-docs-git$
        Show
        Elliot Metsger added a comment - This still seems to be an issue: esm:~/dcs-docs-git$ curl --verbose http://maven.apache.org/doxia/doxia/index.html * About to connect() to maven.apache.org port 80 (#0) * Trying 140.211.11.131... connected * Connected to maven.apache.org (140.211.11.131) port 80 (#0) > GET /doxia/doxia/index.html HTTP/1.1 > User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5 > Host: maven.apache.org > Accept: */* > * Recv failure: Connection reset by peer * Closing connection #0 curl: (56) Recv failure: Connection reset by peer esm:~/dcs-docs-git$
        Hide
        Elliot Metsger added a comment -
        http://status.apache.org/ says that everything is ok (when searching for "www.apache.org", and "140.211.11.131") on the status page.

        This is an intermittent problem that resolves itself in seconds to minutes. I don't know how often status.apache.org is updated, but if the problem occurs and then "fixes itself" before status.apache.org sees a problem...
        Show
        Elliot Metsger added a comment - http://status.apache.org/ says that everything is ok (when searching for " www.apache.org ", and "140.211.11.131") on the status page. This is an intermittent problem that resolves itself in seconds to minutes. I don't know how often status.apache.org is updated, but if the problem occurs and then "fixes itself" before status.apache.org sees a problem...
        Hide
        Elliot Metsger added a comment -
        Right now, my external IP is 128.220.159.25. I'm natted, and I can't guarantee that my external IP is the same all the time, but in general I'd always be coming from 128.220/16. I'm not sure if my IP would show up in the HTTP request logs of 140.211.11.131...
        Show
        Elliot Metsger added a comment - Right now, my external IP is 128.220.159.25. I'm natted, and I can't guarantee that my external IP is the same all the time, but in general I'd always be coming from 128.220/16. I'm not sure if my IP would show up in the HTTP request logs of 140.211.11.131...
        Hide
        Sebb added a comment -
        One of the JMeter tests currently relies on being able to access http://jmeter.apache.org/.
        Jenkins builds are intermittently failing with errors such as

        java.net.SocketException,Non HTTP response message: Connection reset

        May be this is related?
        Show
        Sebb added a comment - One of the JMeter tests currently relies on being able to access http://jmeter.apache.org/ . Jenkins builds are intermittently failing with errors such as java.net.SocketException,Non HTTP response message: Connection reset May be this is related?
        Hide
        Sebb added a comment -
        Also the projects build script has been intermittently reporting errors to the site-dev list, for example the most recent is:

        From: apsite@apache.org (Cron Daemon)
        To: site-dev@apache.org
        Subject: Cron <apsite@minotaur> /x1/home/apsite/wrkdir/bin/build.sh
        Message-Id: <20120817213208.E0956D93B@minotaur.apache.org>
        Date: Fri, 17 Aug 2012 21:30:40 +0000 (UTC)

        projects_21.log:Failed to get 'http://accumulo.apache.org/doap_Accumulo.rdf&#39;
        projects_21.log:Failed to get 'http://ant.apache.org/doap_Ant.rdf&#39;
        projects_21.log:Failed to get 'http://forrest.apache.org/doap.xml&#39;
        projects_21.log:Failed to get 'http://jackrabbit.apache.org/doap_Jackrabbit.rdf&#39;
        projects_21.log:Failed to get 'http://jena.apache.org/about_jena/jena.rdf&#39;
        projects_21.log:Failed to get 'http://lucene.apache.org/core/doap.rdf&#39;
        projects_21.log:Failed to get 'http://lucene.apache.org/solr/doap.rdf&#39;
        projects_21.log:Failed to get 'http://manifoldcf.apache.org/doap_ManifoldCF.rdf&#39;
        projects_21.log:Failed to get 'http://nutch.apache.org/doap.rdf&#39;
        projects_21.log:Failed to get 'http://oodt.apache.org/doap_oodt.rdf&#39;
        projects_21.log:Failed to get 'http://spamassassin.apache.org/doap.rdf&#39;
        projects_21.log:Failed to get 'http://subversion.apache.org/doap.rdf&#39;
        projects_21.log:Failed to get 'http://zookeeper.apache.org/doap.rdf&#39;
        Show
        Sebb added a comment - Also the projects build script has been intermittently reporting errors to the site-dev list, for example the most recent is: From: apsite@apache.org (Cron Daemon) To: site-dev@apache.org Subject: Cron < apsite@minotaur > /x1/home/apsite/wrkdir/bin/build.sh Message-Id: < 20120817213208.E0956D93B@minotaur.apache.org > Date: Fri, 17 Aug 2012 21:30:40 +0000 (UTC) projects_21.log:Failed to get ' http://accumulo.apache.org/doap_Accumulo.rdf&#39; projects_21.log:Failed to get ' http://ant.apache.org/doap_Ant.rdf&#39; projects_21.log:Failed to get ' http://forrest.apache.org/doap.xml&#39; projects_21.log:Failed to get ' http://jackrabbit.apache.org/doap_Jackrabbit.rdf&#39; projects_21.log:Failed to get ' http://jena.apache.org/about_jena/jena.rdf&#39; projects_21.log:Failed to get ' http://lucene.apache.org/core/doap.rdf&#39; projects_21.log:Failed to get ' http://lucene.apache.org/solr/doap.rdf&#39; projects_21.log:Failed to get ' http://manifoldcf.apache.org/doap_ManifoldCF.rdf&#39; projects_21.log:Failed to get ' http://nutch.apache.org/doap.rdf&#39; projects_21.log:Failed to get ' http://oodt.apache.org/doap_oodt.rdf&#39; projects_21.log:Failed to get ' http://spamassassin.apache.org/doap.rdf&#39; projects_21.log:Failed to get ' http://subversion.apache.org/doap.rdf&#39; projects_21.log:Failed to get ' http://zookeeper.apache.org/doap.rdf&#39;
        Hide
        #asfinfra IRC Bot added a comment -
        <danielsh> 1) the one to look at is the HTTP* service on {eos,aurora}.a.o
        Show
        #asfinfra IRC Bot added a comment - <danielsh> 1) the one to look at is the HTTP* service on {eos,aurora}.a.o
        Hide
        #asfinfra IRC Bot added a comment -
        <danielsh> ... and you could have figured this out by reverse DNS lookup against the IP address
        Show
        #asfinfra IRC Bot added a comment - <danielsh> ... and you could have figured this out by reverse DNS lookup against the IP address
        Hide
        #asfinfra IRC Bot added a comment -
        <danielsh> site-dev@ cron hasn't failed since Friday. Assuming fixed. @Elliot no bans matching your IP addr on 140.211.11.131 . Reopen if there is still a problem.
        Show
        #asfinfra IRC Bot added a comment - <danielsh> site-dev@ cron hasn't failed since Friday. Assuming fixed. @Elliot no bans matching your IP addr on 140.211.11.131 . Reopen if there is still a problem.

          People

          • Assignee:
            Unassigned
            Reporter:
            Elliot Metsger
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development