Ivy
  1. Ivy
  2. IVY-1194

Downloads from maven repository fail when using transparent HTTP proxies

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.1.0
    • Fix Version/s: 2.2.0-RC1
    • Component/s: Core
    • Labels:
      None
    • Environment:

      Windows XP, Java 1.5.0.16, java 1.6.0.20

      Description

      The package without dependencies cannot fetch dependencies and samples in the package with dependencies do not work.

      Fetching dependencies fails with the following log:

      Buildfile: build.xml

      init-ivy:

      retrieve-all:
      [ivy:retrieve] :: Ivy 2.1.0 - 20090925235825 :: http://ant.apache.org/ivy/ ::
      [ivy:retrieve] :: loading settings :: url = jar:file:/C:/utils/apache/apache-ivy-2.1.0-wo/ivy.jar!/org/apache/ivy/core/settings/ivysettings.xml
      [ivy:retrieve] :: resolving dependencies :: org.apache.ivy#ivy;2.1.0
      [ivy:retrieve] confs: [core, httpclient, oro, vfs, sftp, standalone, ant, default, test, source]
      [ivy:retrieve] found commons-httpclient#commons-httpclient;3.0 in public
      [ivy:retrieve] found commons-codec#commons-codec;1.2 in public
      [ivy:retrieve] found oro#oro;2.0.8 in public
      [ivy:retrieve] found commons-vfs#commons-vfs;1.0 in public
      [ivy:retrieve] found com.jcraft#jsch;0.1.31 in public
      [ivy:retrieve] found ant#ant;1.6.2 in public
      [ivy:retrieve] found ant#ant-nodeps;1.6.2 in public
      [ivy:retrieve] found ant#ant-trax;1.6.2 in public
      [ivy:retrieve] found junit#junit;3.8.2 in public
      [ivy:retrieve] found commons-lang#commons-lang;2.4 in public
      [ivy:retrieve] found org.apache.ant#ant-testutil;1.7.0 in public
      [ivy:retrieve] found ant#ant-launcher;1.6.2 in public
      [ivy:retrieve] found xerces#xercesImpl;2.6.2 in public
      [ivy:retrieve] found xerces#xmlParserAPIs;2.6.2 in public
      [ivy:retrieve] downloading http://repo1.maven.org/maven2/junit/junit/3.8.2/junit-3.8.2.jar ...
      [ivy:retrieve] ............ (117kB)
      [ivy:retrieve] .. (0kB)
      [ivy:retrieve] :: resolution report :: resolve 1250ms :: artifacts dl 250ms
      [ivy:retrieve] :: evicted modules:
      [ivy:retrieve] commons-logging#commons-logging;1.0.3 by commons-logging#commons-logging;1.0.4 in [default]
      ---------------------------------------------------------------------

        modules artifacts
      conf number search dwnlded evicted number dwnlded

      ---------------------------------------------------------------------

      core 0 0 0 0 0 0
      httpclient 3 0 0 0 2 0
      oro 1 0 0 0 1 0
      vfs 2 0 0 0 1 0
      sftp 1 0 0 0 1 0
      standalone 0 0 0 0 0 0
      ant 1 0 0 0 1 0
      default 10 0 0 1 8 0
      test 6 0 0 0 6 0
      source 0 0 0 0 0 0

      ---------------------------------------------------------------------

      [ivy:retrieve] :: problems summary ::
      [ivy:retrieve] :::: WARNINGS
      [ivy:retrieve] problem while downloading module descriptor: http://repo1.maven.org/maven2/commons-logging/commons-logging/1.0.3/commons-logging-1.0.3.pom: invalid sha1: expected=<
      [ivy:retrieve] г computed=88c58ea4a562116ab15fb76c9097ee1f25cc750b (62ms)
      [ivy:retrieve] module not found: commons-logging#commons-logging;1.0.3
      [ivy:retrieve] ==== local: tried
      [ivy:retrieve] C:\Documents and Settings\Constantin.Plotnikov\.ivy2/local/commons-logging/commons-logging/1.0.3/ivys/ivy.xml
      [ivy:retrieve] – artifact commons-logging#commons-logging;1.0.3!commons-logging.jar:
      [ivy:retrieve] C:\Documents and Settings\Constantin.Plotnikov\.ivy2/local/commons-logging/commons-logging/1.0.3/jars/commons-logging.jar
      [ivy:retrieve] ==== shared: tried
      [ivy:retrieve] C:\Documents and Settings\Constantin.Plotnikov\.ivy2/shared/commons-logging/commons-logging/1.0.3/ivys/ivy.xml
      [ivy:retrieve] – artifact commons-logging#commons-logging;1.0.3!commons-logging.jar:
      [ivy:retrieve] C:\Documents and Settings\Constantin.Plotnikov\.ivy2/shared/commons-logging/commons-logging/1.0.3/jars/commons-logging.jar
      [ivy:retrieve] ==== public: tried
      [ivy:retrieve] http://repo1.maven.org/maven2/commons-logging/commons-logging/1.0.3/commons-logging-1.0.3.pom
      [ivy:retrieve] problem while downloading module descriptor: http://repo1.maven.org/maven2/commons-logging/commons-logging/1.0.4/commons-logging-1.0.4.pom: invalid sha1: expected=<
      [ivy:retrieve] module not found: commons-logging#commons-logging;1.0.4
      [ivy:retrieve] ==== local: tried
      [ivy:retrieve] C:\Documents and Settings\Constantin.Plotnikov\.ivy2/local/commons-logging/commons-logging/1.0.4/ivys/ivy.xml
      [ivy:retrieve] – artifact commons-logging#commons-logging;1.0.4!commons-logging.jar:
      [ivy:retrieve] C:\Documents and Settings\Constantin.Plotnikov\.ivy2/local/commons-logging/commons-logging/1.0.4/jars/commons-logging.jar
      [ivy:retrieve] ==== shared: tried
      [ivy:retrieve] C:\Documents and Settings\Constantin.Plotnikov\.ivy2/shared/commons-logging/commons-logging/1.0.4/ivys/ivy.xml
      [ivy:retrieve] – artifact commons-logging#commons-logging;1.0.4!commons-logging.jar:
      [ivy:retrieve] C:\Documents and Settings\Constantin.Plotnikov\.ivy2/shared/commons-logging/commons-logging/1.0.4/jars/commons-logging.jar
      [ivy:retrieve] ==== public: tried
      [ivy:retrieve] http://repo1.maven.org/maven2/commons-logging/commons-logging/1.0.4/commons-logging-1.0.4.pom
      [ivy:retrieve] [FAILED ] junit#junit;3.8.2!junit.jar: invalid sha1: expected=<
      [ivy:retrieve] [FAILED ] junit#junit;3.8.2!junit.jar: (0ms)
      [ivy:retrieve] ==== shared: tried
      [ivy:retrieve] C:\Documents and Settings\Constantin.Plotnikov\.ivy2/shared/junit/junit/3.8.2/jars/junit.jar
      [ivy:retrieve] ==== public: tried
      [ivy:retrieve] http://repo1.maven.org/maven2/junit/junit/3.8.2/junit-3.8.2.jar
      [ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
      [ivy:retrieve] :: UNRESOLVED DEPENDENCIES ::
      [ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
      [ivy:retrieve] :: commons-logging#commons-logging;1.0.3: not found
      [ivy:retrieve] :: commons-logging#commons-logging;1.0.4: not found
      [ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
      [ivy:retrieve]
      [ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
      [ivy:retrieve] :: FAILED DOWNLOADS ::
      [ivy:retrieve] :: ^ see resolution messages for details ^ ::
      [ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
      [ivy:retrieve] :: junit#junit;3.8.2!junit.jar
      [ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
      [ivy:retrieve]
      [ivy:retrieve]
      [ivy:retrieve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS

      The sample in package with dependencies fails with the following log:

      Buildfile: build.xml

      download-ivy:
      [mkdir] Created dir: C:\utils\apache\apache-ivy-2.1.0\src\example\go-ivy\ivy
      [echo] installing ivy...
      [get] Getting: http://repo1.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar
      [get] To: C:\utils\apache\apache-ivy-2.1.0\src\example\go-ivy\ivy\ivy.jar

      install-ivy:
      [taskdef] java.util.zip.ZipException: error in opening zip file
      [taskdef] at java.util.zip.ZipFile.open(Native Method)
      [taskdef] at java.util.zip.ZipFile.<init>(ZipFile.java:203)
      [taskdef] at java.util.zip.ZipFile.<init>(ZipFile.java:234)
      [taskdef] at org.apache.tools.ant.AntClassLoader.getResourceURL(AntClassLoader.java:919)
      [taskdef] at org.apache.tools.ant.AntClassLoader$ResourceEnumeration.findNextResource(AntClassLoader.java:126)
      [taskdef] at org.apache.tools.ant.AntClassLoader$ResourceEnumeration.<init>(AntClassLoader.java:88)
      [taskdef] at org.apache.tools.ant.AntClassLoader.findResources(AntClassLoader.java:869)
      [taskdef] at java.lang.ClassLoader.getResources(ClassLoader.java:1015)
      [taskdef] at org.apache.tools.ant.taskdefs.Definer.resourceToURLs(Definer.java:267)
      [taskdef] at org.apache.tools.ant.taskdefs.Definer.execute(Definer.java:211)
      [taskdef] at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:275)
      [taskdef] at org.apache.tools.ant.Task.perform(Task.java:364)
      [taskdef] at org.apache.tools.ant.Target.execute(Target.java:341)
      [taskdef] at org.apache.tools.ant.Target.performTasks(Target.java:369)
      [taskdef] at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1216)
      [taskdef] at org.apache.tools.ant.Project.executeTarget(Project.java:1185)
      [taskdef] at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:40)
      [taskdef] at org.apache.tools.ant.Project.executeTargets(Project.java:1068)
      [taskdef] at org.apache.tools.ant.Main.runBuild(Main.java:668)
      [taskdef] at org.apache.tools.ant.Main.startAnt(Main.java:187)
      [taskdef] at org.apache.tools.ant.launch.Launcher.run(Launcher.java:246)
      [taskdef] at org.apache.tools.ant.launch.Launcher.main(Launcher.java:67)
      [taskdef] Could not load definitions from resource org/apache/ivy/ant/antlib.xml. It could not be found.

      generate-src:

      go:
      [echo] using ivy to resolve commons-lang 2.1...

      BUILD FAILED
      C:\utils\apache\apache-ivy-2.1.0\src\example\go-ivy\build.xml:85: Could not create task or type of type: antlib:org.apache.ivy.ant:cachepath.

      Ant could not find the task or a class this task relies upon.

      This is common and has a number of causes; the usual
      solutions are to read the manual pages then download and
      install needed JAR files, or fix the build file:

      • You have misspelt 'antlib:org.apache.ivy.ant:cachepath'.
        Fix: check your spelling.
      • The task needs an external JAR file to execute
        and this is not found at the right place in the classpath.
        Fix: check the documentation for dependencies.
        Fix: declare the task.
      • The task is an Ant optional task and the JAR file and/or libraries
        implementing the functionality were not found at the time you
        yourself built your installation of Ant from the Ant sources.
        Fix: Look in the ANT_HOME/lib for the 'ant-' JAR corresponding to the
        task and make sure it contains more than merely a META-INF/MANIFEST.MF.
        If all it contains is the manifest, then rebuild Ant with the needed
        libraries present in $
        Unknown macro: {ant.home}

        /lib/optional/ , or alternatively,
        download a pre-built release version from apache.org

      • The build file was written for a later version of Ant
        Fix: upgrade to at least the latest release version of Ant
      • The task is not an Ant core or optional task
        and needs to be declared using <taskdef>.
      • You are attempting to use a task defined using
        <presetdef> or <macrodef> but have spelt wrong or not
        defined it at the point of use

      Remember that for JAR files to be visible to Ant tasks implemented
      in ANT_HOME/lib, the files must be in the same directory or on the
      classpath

      Please neither file bug reports on this problem, nor email the
      Ant mailing lists, until all of these causes have been explored,
      as this is not an Ant bug.

      Total time: 1 second

      The reason for the bug is the squid. When request comes and it has a cached content with gzip content encoding, it return a content with encoded content, event if no encoding was specified in the request or "Accept-Encoding: identity" was specified. The below is a sample log that demonstrates the problem:

      DEBUG output created by Wget 1.11.4 on cygwin.

      -2010-06-02 12:41:55- http://repo1.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar
      Resolving repo1.maven.org... 38.97.124.18
      Caching repo1.maven.org => 38.97.124.18
      Connecting to repo1.maven.org|38.97.124.18|:80... connected.
      Created socket 4.
      Releasing 0x006d9100 (new refcount 1).

      --request begin--
      GET /maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar HTTP/1.0
      User-Agent: Wget/1.11.4
      Accept: /
      Host: repo1.maven.org
      Connection: Keep-Alive

      --request end--
      HTTP request sent, awaiting response...
      --response begin--
      HTTP/1.0 200 OK
      Server: nginx/0.7.62
      Date: Tue, 01 Jun 2010 13:09:36 GMT
      Content-Type: application/java-archive
      Last-Modified: Mon, 12 Oct 2009 21:19:02 GMT
      Content-Encoding: gzip
      X-Cache: MISS from squid_havp_node1
      X-Cache-Lookup: MISS from squid_havp_node1:3128
      Age: 70339
      Content-Length: 834513
      X-Cache: HIT from squid_havp_node1
      X-Cache-Lookup: HIT from squid_havp_node1:3128
      Connection: keep-alive

      --response end--

      HTTP/1.0 200 OK
      Server: nginx/0.7.62
      Date: Tue, 01 Jun 2010 13:09:36 GMT
      Content-Type: application/java-archive
      Last-Modified: Mon, 12 Oct 2009 21:19:02 GMT
      Content-Encoding: gzip
      X-Cache: MISS from squid_havp_node1
      X-Cache-Lookup: MISS from squid_havp_node1:3128
      Age: 70339
      Content-Length: 834513
      X-Cache: HIT from squid_havp_node1
      X-Cache-Lookup: HIT from squid_havp_node1:3128
      Connection: keep-alive
      Registered socket 4 for persistent reuse.
      Length: 834513 (815K) [application/java-archive]
      Saving to: `ivy-2.1.0.jar'

      0K .......... .......... .......... .......... .......... 6% 8.49M 0s
      50K .......... .......... .......... .......... .......... 12% 12.0M 0s
      100K .......... .......... .......... .......... .......... 18% 7.26M 0s
      150K .......... .......... .......... .......... .......... 24% 9.28M 0s
      200K .......... .......... .......... .......... .......... 30% 5.33M 0s
      250K .......... .......... .......... .......... .......... 36% 9.31M 0s
      300K .......... .......... .......... .......... .......... 42% 5.26M 0s
      350K .......... .......... .......... .......... .......... 49% 9.15M 0s
      400K .......... .......... .......... .......... .......... 55% 7.27M 0s
      450K .......... .......... .......... .......... .......... 61% 7.42M 0s
      500K .......... .......... .......... .......... .......... 67% 9.21M 0s
      550K .......... .......... .......... .......... .......... 73% 6.11M 0s
      600K .......... .......... .......... .......... .......... 79% 9.50M 0s
      650K .......... .......... .......... .......... .......... 85% 8.84M 0s
      700K .......... .......... .......... .......... .......... 92% 9.54M 0s
      750K .......... .......... .......... .......... .......... 98% 6.04M 0s
      800K .......... .... 100% 16.2K=1.0s

      2010-06-02 12:41:56 (795 KB/s) - `ivy-2.1.0.jar' saved [834513/834513]

      In our environment, a bunch of files from repo1.maven.org are cached in the squid. So there were a lot of conflicts with content encoding.

      The possible workarounds:
      1. Check returned content encoding from http client, if it is not identity, reissue the request with disabling the caches.
      2. Ungzip files fetched with gzip content encoding.
      3. Support gzip content encoding in common HTTP.
      3. Use local maven proxies

        Issue Links

          Activity

            People

            • Assignee:
              Maarten Coene
              Reporter:
              Constantine Plotnikov
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development