Ivy
  1. Ivy
  2. IVY-1179

Packager resolver always extracts all files from archives even when the packaging instructions contains include tags

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.1.0, 2.3.0-RC1
    • Fix Version/s: 2.2.0-RC1
    • Component/s: Core

      Description

      According to the documentation the packager resolver should unzip/untar only specific files and folders (and not the complete archive) from downloaded resources in case the packager module description specfies include tags. Let's assume we have a module testng (example is taken from the examples in the Packager Resolver documentation) and we have a packager module descriptor (the packaging instructions) packager.xml specifying the following resource:

      <resource dest="archive" url="http://testng.org/${zipname}.zip" sha1="2ea19275dc17453306f8bb780fe6ef6e9af7756b">
              <include name="${zipname}/*.jar"/>
      </resource>
      

      this instructs the package resolver to download the resources from http://testng.org/... and then extract the archive, but only all JAR files in the root of the archive.

      in this case only the JAR's in the ZIP file should be extracted from the resource, but that's not the case. at first i didn't notice this, but i was creating the packaging instructions for a project which is in a huge ZIP file and so i only wanted to extract the files that i actually needed, but i noticed that the extraction took way to long.

      i then configured the packager resolver to preserve the build directories (where the archive is extracted) for debugging purposes and i noticed that the complete archive was still extracted although the packager.xml file contained the necessary 'include' tags to only inluce 3 JAR's.

      when using the preserveBuildDirectories switch on the packager resolver, it's also possible to view the ANT build file packager-output.xml (this one is generated by the packager resolver based on the packaging instructions). this build file is executed by the package resolver to do the extraction process. this is the relevant part of the packager-output.xml file for the above mentioded example for TestNG:

      <unzip src="${resdir.N65541}${file.separator}${filename.N65541}" dest="archive">
          <fileset dir=".">
              <include name="testng-2.5/*.jar"/>
           </fileset>
      </unzip>
      

      but this is incorrect and that's why the complete zip is still extracted. the 'fileset' tag should be used to indicate a set of ZIP files that need to be extracted, not to indicate which resources from the archive (specified in the 'src' attribute of the unzip tag) need to be included when extracting. 'patternset' is what we need here instead of 'fileset'

      the actual problem is situated in the file src/java/org/apache/ivy/plugins/resolver/packager/packager.xsl on line 420 (in the ivy core project).

      <xsl:when test="$type = 'zip' or $type = 'war' or $type = 'jar'">
          <unzip src="{$file}" dest="{$dir}">
              <xsl:if test="$includes">
                  <fileset dir=".">
                      <xsl:copy-of select="$includes"/>
                  </fileset>
              </xsl:if>
          </unzip>
      </xsl:when>
      

      should be:

      <xsl:when test="$type = 'zip' or $type = 'war' or $type = 'jar'">
          <unzip src="{$file}" dest="{$dir}">
              <xsl:if test="$includes">
                  <patternset>
                      <xsl:copy-of select="$includes"/>
                  </patternset>
              </xsl:if>
          </unzip>
      </xsl:when>
      

      the same needs to be done for the TAR part in the packages.xsl file.

      i already tried this fix and it works.

      if that's OK, i'll create a patch for this and provide the necessary unit tests to illustrate the problem and the fix.

      1. IVY-1179-unittests.zip
        6 kB
        Stefan De Boey
      2. IVY-1179.patch
        1 kB
        Stefan De Boey
      3. ASF.LICENSE.NOT.GRANTED--IVY-1179-unittests.zip
        6 kB
        Stefan De Boey
      4. ASF.LICENSE.NOT.GRANTED--IVY-1179.patch
        1 kB
        Stefan De Boey

        Activity

        Hide
        Stefan De Boey added a comment -

        i've created an extra example to illustrate the problem. in this example i retrieve the module commons-configuration for company org.apache via the package resolver. the packager resolver is configured to look for the ivy.xml and the packaging instructions in a directory on disk.

        the build.xml file is trivial:

        <project xmlns:ivy="antlib:org.apache.ivy.ant" name="hello-ivy" default="run">
        
            <target name="resolve" description="--> retrieve dependencies with ivy">
            	<ivy:retrieve organisation="org.apache" module="commons-configuration" revision="1.6" inline="true" pattern="./[artifact]-[revision].[ext]"/>
            </target>
        
        </project>
        

        this tells ivy to retrieve the module commons-configuration (version 1.6) via the default resolver.

        the ivysettings.xml:

        <ivysettings>
        
        	<settings defaultResolver="package" />
        
        	<resolvers>
        
        		<packager name="package" buildRoot="${basedir}/packager/build"
        			resourceCache="${basedir}/packager/cache" preserveBuildDirectories="true">
        			<ivy pattern="file://${basedir}/repo/[organisation]/[module]/ivy-[revision].xml" />
        			<artifact pattern="file://${basedir}/repo/[organisation]/[module]/packager-[revision].xml" />
        		</packager>
        
        	</resolvers>
        
        </ivysettings>
        
        

        this specifies a packager resolver as the default resolver. the packager resolver will look for the module configuration file (ivy.xml and packager.xml} in the 'repo' directory.

        the ivy file (ivy-1.6.xml):

        <?xml version="1.0" encoding="UTF-8"?>
        <ivy-module version="2.0"
        	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        	xsi:noNamespaceSchemaLocation="http://ant.apache.org/ivy/schemas/ivy.xsd">
        
        	<info organisation="org.apache" module="commons-configuration" revision="1.6" />
        
        	<publications>
        		<artifact name="commons-configuration" />
        	</publications>
        
        </ivy-module>
        

        and the packaging instruction (packager-1.6.xml):

        <packager-module version="1.0">
        
        	<resource dest="archive"
        		url="http://www.apache.org/dist/commons/configuration/binaries/commons-configuration-1.6.zip"
        		sha1="f716c88e10a9a776fa70815c44ef4ba5781a2a7e">
        
        		<include name="commons-configuration-1.6/commons-configuration-1.6.jar" />
        
        	</resource>
        
        	<build>
        		<move file="archive/commons-configuration-1.6/commons-configuration-1.6.jar" tofile="artifacts/jars/commons-configuration.jar" />
        	</build>
        
        </packager-module>
        
        

        Notice the 'include' tag that specifies to only include (extract) the file commons-configuration-1.6.jar from the commons-configuration directory in the archive (resource)

        Now, if you execute the ANT target and then take a look at the ./packager/build/org.apache/commons-configuration/1.6/archive directory you will see that the complete archive was extracted and not only the commons-configuration-1.6.jar file.

        Show
        Stefan De Boey added a comment - i've created an extra example to illustrate the problem. in this example i retrieve the module commons-configuration for company org.apache via the package resolver. the packager resolver is configured to look for the ivy.xml and the packaging instructions in a directory on disk. the build.xml file is trivial: <project xmlns:ivy = "antlib:org.apache.ivy.ant" name= "hello-ivy" default= "run" > <target name= "resolve" description= "--> retrieve dependencies with ivy" > <ivy:retrieve organisation= "org.apache" module= "commons-configuration" revision= "1.6" inline= "true" pattern= "./[artifact]-[revision].[ext]" /> </target> </project> this tells ivy to retrieve the module commons-configuration (version 1.6) via the default resolver. the ivysettings.xml: <ivysettings> <settings defaultResolver= "package" /> <resolvers> <packager name= "package" buildRoot= "${basedir}/packager/build" resourceCache= "${basedir}/packager/cache" preserveBuildDirectories= "true" > <ivy pattern= "file://${basedir}/repo/[organisation]/[module]/ivy-[revision].xml" /> <artifact pattern= "file://${basedir}/repo/[organisation]/[module]/packager-[revision].xml" /> </packager> </resolvers> </ivysettings> this specifies a packager resolver as the default resolver. the packager resolver will look for the module configuration file (ivy.xml and packager.xml} in the 'repo' directory. the ivy file (ivy-1.6.xml): <?xml version= "1.0" encoding= "UTF-8" ?> <ivy-module version= "2.0" xmlns:xsi = "http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation= "http://ant.apache.org/ivy/schemas/ivy.xsd" > <info organisation= "org.apache" module= "commons-configuration" revision= "1.6" /> <publications> <artifact name= "commons-configuration" /> </publications> </ivy-module> and the packaging instruction (packager-1.6.xml): <packager-module version= "1.0" > <resource dest= "archive" url= "http://www.apache.org/dist/commons/configuration/binaries/commons-configuration-1.6.zip" sha1= "f716c88e10a9a776fa70815c44ef4ba5781a2a7e" > <include name= "commons-configuration-1.6/commons-configuration-1.6.jar" /> </resource> <build> <move file= "archive/commons-configuration-1.6/commons-configuration-1.6.jar" tofile= "artifacts/jars/commons-configuration.jar" /> </build> </packager-module> Notice the 'include' tag that specifies to only include (extract) the file commons-configuration-1.6.jar from the commons-configuration directory in the archive (resource) Now, if you execute the ANT target and then take a look at the ./packager/build/org.apache/commons-configuration/1.6/archive directory you will see that the complete archive was extracted and not only the commons-configuration-1.6.jar file.
        Hide
        Maarten Coene added a comment -

        Your suggested fix seems ok to me.
        Please go ahead and attach your patch + junit tests

        Maarten

        Show
        Maarten Coene added a comment - Your suggested fix seems ok to me. Please go ahead and attach your patch + junit tests Maarten
        Hide
        Stefan De Boey added a comment -

        i've attached a patch that contains a fix to the XSL (packager.xsl) used by the package resolver.

        i also attached a zip file containing 2 tests (in Ivy1179Test class) that illustrate the problem. one test for testing zip files and or for tar files. both test check for the existence of a README file. the test fails if this file was extracted because it shouldn't be. it should be excluded from extraction, only the jar files should be extracted. this test fails if executed on trunk.

        Show
        Stefan De Boey added a comment - i've attached a patch that contains a fix to the XSL (packager.xsl) used by the package resolver. i also attached a zip file containing 2 tests (in Ivy1179Test class) that illustrate the problem. one test for testing zip files and or for tar files. both test check for the existence of a README file. the test fails if this file was extracted because it shouldn't be. it should be excluded from extraction, only the jar files should be extracted. this test fails if executed on trunk.
        Hide
        Maarten Coene added a comment -

        Thanks for the patch! Could you re-attach your files and check the "grant license to ASF..." option?

        Show
        Maarten Coene added a comment - Thanks for the patch! Could you re-attach your files and check the "grant license to ASF..." option?
        Hide
        Maarten Coene added a comment -

        I've committed your patch + junit tests (with 1 minor modification to get them working on JDK 1.4).
        Thanks a lot for the contribution!

        Show
        Maarten Coene added a comment - I've committed your patch + junit tests (with 1 minor modification to get them working on JDK 1.4). Thanks a lot for the contribution!

          People

          • Assignee:
            Maarten Coene
            Reporter:
            Stefan De Boey
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development