Lucene - Core
  1. Lucene - Core
  2. LUCENE-4354

add validate-maven task to check maven dependencies

    Details

    • Type: Task Task
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      We had a situation where the maven artifacts depended on the wrong version of tika: we should test that the maven dependencies are correct.

      An easy way to do this is to force it to download all of its dependencies, and then run our existing license checks over that.

      This currently fails: maven is bringing in some extra 3rd party libraries.

      1. LUCENE-4354_hacked_lucene_only.patch
        6 kB
        Robert Muir
      2. LUCENE-4354.patch
        14 kB
        Robert Muir
      3. LUCENE-4354.patch
        14 kB
        Robert Muir
      4. LUCENE-4354.patch
        13 kB
        Robert Muir
      5. LUCENE-4354.patch
        13 kB
        Uwe Schindler
      6. LUCENE-4354.patch
        13 kB
        Uwe Schindler
      7. LUCENE-4354.patch
        9 kB
        Robert Muir
      8. LUCENE-4354.patch
        17 kB
        Robert Muir
      9. LUCENE-4354.patch
        16 kB
        Robert Muir
      10. LUCENE-4354.patch
        13 kB
        Robert Muir
      11. LUCENE-4354.patch
        6 kB
        Robert Muir
      12. LUCENE-4354.patch
        3 kB
        Robert Muir
      13. LUCENE-4354-dep-fix.patch
        7 kB
        selckin

        Activity

        Hide
        Robert Muir added a comment -

        here's a prototype: it should really be using the maven-ant-tasks plugin and not 'exec' but its a start.

        Show
        Robert Muir added a comment - here's a prototype: it should really be using the maven-ant-tasks plugin and not 'exec' but its a start.
        Hide
        Robert Muir added a comment -

        lucene/ fails because maven is pulling in an extra jar (duplicates removed below):

        -maven-check-licenses:
             [echo] License check under: /home/rmuir/workspace/lucene-trunk/lucene/../maven-build/lucene
         [licenses] MISSING sha1 checksum file for: /home/rmuir/workspace/lucene-trunk/maven-build/lucene/analysis/common/target/dependency/hamcrest-core-1.1.jar
        ...
        

        solr/ fails due to several extra jars (duplicates removed below):

         [licenses] MISSING sha1 checksum file for: /home/rmuir/workspace/lucene-trunk/maven-build/solr/contrib/analysis-extras/target/dependency/commons-logging-1.1.1.jar
         [licenses] MISSING sha1 checksum file for: /home/rmuir/workspace/lucene-trunk/maven-build/solr/contrib/analysis-extras/target/dependency/hamcrest-core-1.1.jar
         [licenses] MISSING sha1 checksum file for: /home/rmuir/workspace/lucene-trunk/maven-build/solr/contrib/analysis-extras/target/dependency/javax.servlet-3.0.0.v201112011016.jar
         [licenses] MISSING sha1 checksum file for: /home/rmuir/workspace/lucene-trunk/maven-build/solr/contrib/analysis-extras/target/dependency/servlet-api-2.4.jar
         [licenses] MISSING sha1 checksum file for: /home/rmuir/workspace/lucene-trunk/maven-build/solr/contrib/clustering/target/dependency/ehcache-core-1.7.2.jar
         [licenses] MISSING sha1 checksum file for: /home/rmuir/workspace/lucene-trunk/maven-build/solr/contrib/clustering/target/dependency/jdom-1.1.jar
         [licenses] MISSING sha1 checksum file for: /home/rmuir/workspace/lucene-trunk/maven-build/solr/contrib/clustering/target/dependency/morfologik-stemming-1.5.1.jar
         [licenses] MISSING sha1 checksum file for: /home/rmuir/workspace/lucene-trunk/maven-build/solr/contrib/clustering/target/dependency/rome-1.0.0.jar
         [licenses] MISSING sha1 checksum file for: /home/rmuir/workspace/lucene-trunk/maven-build/solr/contrib/clustering/target/dependency/rome-fetcher-1.0.0.jar
        
        Show
        Robert Muir added a comment - lucene/ fails because maven is pulling in an extra jar (duplicates removed below): -maven-check-licenses: [echo] License check under: /home/rmuir/workspace/lucene-trunk/lucene/../maven-build/lucene [licenses] MISSING sha1 checksum file for: /home/rmuir/workspace/lucene-trunk/maven-build/lucene/analysis/common/target/dependency/hamcrest-core-1.1.jar ... solr/ fails due to several extra jars (duplicates removed below): [licenses] MISSING sha1 checksum file for: /home/rmuir/workspace/lucene-trunk/maven-build/solr/contrib/analysis-extras/target/dependency/commons-logging-1.1.1.jar [licenses] MISSING sha1 checksum file for: /home/rmuir/workspace/lucene-trunk/maven-build/solr/contrib/analysis-extras/target/dependency/hamcrest-core-1.1.jar [licenses] MISSING sha1 checksum file for: /home/rmuir/workspace/lucene-trunk/maven-build/solr/contrib/analysis-extras/target/dependency/javax.servlet-3.0.0.v201112011016.jar [licenses] MISSING sha1 checksum file for: /home/rmuir/workspace/lucene-trunk/maven-build/solr/contrib/analysis-extras/target/dependency/servlet-api-2.4.jar [licenses] MISSING sha1 checksum file for: /home/rmuir/workspace/lucene-trunk/maven-build/solr/contrib/clustering/target/dependency/ehcache-core-1.7.2.jar [licenses] MISSING sha1 checksum file for: /home/rmuir/workspace/lucene-trunk/maven-build/solr/contrib/clustering/target/dependency/jdom-1.1.jar [licenses] MISSING sha1 checksum file for: /home/rmuir/workspace/lucene-trunk/maven-build/solr/contrib/clustering/target/dependency/morfologik-stemming-1.5.1.jar [licenses] MISSING sha1 checksum file for: /home/rmuir/workspace/lucene-trunk/maven-build/solr/contrib/clustering/target/dependency/rome-1.0.0.jar [licenses] MISSING sha1 checksum file for: /home/rmuir/workspace/lucene-trunk/maven-build/solr/contrib/clustering/target/dependency/rome-fetcher-1.0.0.jar
        Hide
        Chris Male added a comment -

        hamcrest is a transitive dependency of junit, we'll need to exclude that specifically in our poms.

        Show
        Chris Male added a comment - hamcrest is a transitive dependency of junit, we'll need to exclude that specifically in our poms.
        Hide
        Robert Muir added a comment -

        Yeah some of this might be false fails. i just wanted to get it up here as a prototype.

        I think its a good check we could add to e.g. our nightly jenkins runs once we get it sorted out to prevent problems.

        Its true running the tests with maven is good: but this would fail in a more obvious way (e.g. the tika situation),
        and wouldn't rely upon having tests that ensure all dependencies are correct (I know we don't have this today!)

        Show
        Robert Muir added a comment - Yeah some of this might be false fails. i just wanted to get it up here as a prototype. I think its a good check we could add to e.g. our nightly jenkins runs once we get it sorted out to prevent problems. Its true running the tests with maven is good: but this would fail in a more obvious way (e.g. the tika situation), and wouldn't rely upon having tests that ensure all dependencies are correct (I know we don't have this today!)
        Hide
        selckin added a comment -

        If you add '-DincludeScope=runtime' to the copy call, quite a few disappear, one hamcrest still there, probably from test-framework ?
        Also "ant clean" doesn't remove the copied jars

        Show
        selckin added a comment - If you add '-DincludeScope=runtime' to the copy call, quite a few disappear, one hamcrest still there, probably from test-framework ? Also "ant clean" doesn't remove the copied jars
        Hide
        Robert Muir added a comment -

        I think any dependencies should be checked? I don't think we should hide ones only used
        for test-framework or whatever (its still a maven artifact published out there, unless that is intended to change).

        'ant clean' doesnt remove anything maven-related: there is a maven-related clean task (clean-maven-build)
        that does that, I think its documented in the README.maven

        Show
        Robert Muir added a comment - I think any dependencies should be checked? I don't think we should hide ones only used for test-framework or whatever (its still a maven artifact published out there, unless that is intended to change). 'ant clean' doesnt remove anything maven-related: there is a maven-related clean task (clean-maven-build) that does that, I think its documented in the README.maven
        Hide
        selckin added a comment -

        Well only things from the runtime scope will end up in the war, and i thought i saw other issues where the policy was to ignore license and other checks for build only jars. But i guess it can't hurt to force all the deps to be the same, so ignore me

        Show
        selckin added a comment - Well only things from the runtime scope will end up in the war, and i thought i saw other issues where the policy was to ignore license and other checks for build only jars. But i guess it can't hurt to force all the deps to be the same, so ignore me
        Hide
        Chris Male added a comment -

        Ignoring the scope issue, the validation has revealed valid issues. For example the jdom, rome and servlet dependencies all have different versions to our license files.

        Show
        Chris Male added a comment - Ignoring the scope issue, the validation has revealed valid issues. For example the jdom, rome and servlet dependencies all have different versions to our license files.
        Hide
        Chris Male added a comment -

        It's not just an issue of what ends up in the war since we also publish individual artifacts / poms.

        Show
        Chris Male added a comment - It's not just an issue of what ends up in the war since we also publish individual artifacts / poms.
        Hide
        Robert Muir added a comment -

        Right, the check here is the same set of license exclusions etc used for the 'normal artifacts' that are put on the apache download site.

        Maven shouldnt have any extra dependencies beyond that.

        Show
        Robert Muir added a comment - Right, the check here is the same set of license exclusions etc used for the 'normal artifacts' that are put on the apache download site. Maven shouldnt have any extra dependencies beyond that.
        Hide
        Uwe Schindler added a comment -

        There is a trick with maven-ant-tasks to get a refid to a <fileset/> with all maven dependencies (something like that, maybe Steven knows how to set this up):

        <maven:dependencies xmlns:maven="urn:maven-artifact-ant" filesetId="foobar.fileset" useScope="runtime"/>
        

        After that the ANT fileset should point (like ivy:cachefilset) to all dependency jars. It should be easy to run the license checker on top of this fileset (refid="foobar.fileset").

        See also for a code example: http://ptrthomas.wordpress.com/2009/03/08/why-you-should-use-the-maven-ant-tasks-instead-of-maven-or-ivy/

        Show
        Uwe Schindler added a comment - There is a trick with maven-ant-tasks to get a refid to a <fileset/> with all maven dependencies (something like that, maybe Steven knows how to set this up): <maven:dependencies xmlns:maven = "urn:maven-artifact-ant" filesetId= "foobar.fileset" useScope= "runtime" /> After that the ANT fileset should point (like ivy:cachefilset) to all dependency jars. It should be easy to run the license checker on top of this fileset (refid="foobar.fileset"). See also for a code example: http://ptrthomas.wordpress.com/2009/03/08/why-you-should-use-the-maven-ant-tasks-instead-of-maven-or-ivy/
        Hide
        Robert Muir added a comment -

        Thanks Uwe... this looks like the way to go to remove the mvn.exe hack

        Show
        Robert Muir added a comment - Thanks Uwe... this looks like the way to go to remove the mvn.exe hack
        Hide
        Robert Muir added a comment -

        I hacked this up a little bit for lucene/ to try to use the maven-ant-tasks fileset as Uwe suggested.

        It doesn't totally work: it seems to only be looking at lucene-core but not working 'recursively' on all the lucene modules.

        I didnt touch solr yet.

        Show
        Robert Muir added a comment - I hacked this up a little bit for lucene/ to try to use the maven-ant-tasks fileset as Uwe suggested. It doesn't totally work: it seems to only be looking at lucene-core but not working 'recursively' on all the lucene modules. I didnt touch solr yet.
        Hide
        selckin added a comment -

        fixes some deps (ran with runtime scope)

        • servlet-api is confusing me
        • warns about common-compress in some solr contribs but it is the correct version
        Show
        selckin added a comment - fixes some deps (ran with runtime scope) servlet-api is confusing me warns about common-compress in some solr contribs but it is the correct version
        Hide
        Uwe Schindler added a comment -

        Robert: Maybe we do the check per module? We can do this after the create-maven-artifacts module target.

        Show
        Uwe Schindler added a comment - Robert: Maybe we do the check per module? We can do this after the create-maven-artifacts module target.
        Hide
        Robert Muir added a comment -

        Good idea Uwe! Here is a new patch doing that.

        Show
        Robert Muir added a comment - Good idea Uwe! Here is a new patch doing that.
        Hide
        Robert Muir added a comment -

        Thanks selckin: I think I will let Steve or someone more familiar review your patch.

        About the commons-compress: there is a problem somewhere. Some things are depending on 1.2 and others on 1.3.
        But nothing should be depending on 1.2: Thats causing the failure.

        Show
        Robert Muir added a comment - Thanks selckin: I think I will let Steve or someone more familiar review your patch. About the commons-compress: there is a problem somewhere. Some things are depending on 1.2 and others on 1.3. But nothing should be depending on 1.2: Thats causing the failure.
        Hide
        Uwe Schindler added a comment -

        Looks fine, works almost on windows, stupidity is the <restrict> thing that matches also on the file separator:

        <rsel:not>
         <rsel:or>
          <rsel:name name="**/lucene-*.jar" handledirsep="true"/>
          <rsel:name name="**/solr-*.jar" handledirsep="true"/>
         </rsel:or>
        </rsel:not>
        

        The handledirsep="true" is the needed magic. Otherwise looks fine, only some dependnecies are wrong.

        Show
        Uwe Schindler added a comment - Looks fine, works almost on windows, stupidity is the <restrict> thing that matches also on the file separator: <rsel:not> <rsel:or> <rsel:name name= "**/lucene-*.jar" handledirsep= "true" /> <rsel:name name= "**/solr-*.jar" handledirsep= "true" /> </rsel:or> </rsel:not> The handledirsep="true" is the needed magic. Otherwise looks fine, only some dependnecies are wrong.
        Hide
        Robert Muir added a comment -

        Updated patch:

        • added workCorrectlyOnWindows=true as Uwe suggested
        • incorporated selckin's patch.

        its now passing on lucene/. I will investigate solr/ now (not tested yet)

        Show
        Robert Muir added a comment - Updated patch: added workCorrectlyOnWindows=true as Uwe suggested incorporated selckin's patch. its now passing on lucene/. I will investigate solr/ now (not tested yet)
        Hide
        selckin added a comment -

        Since transitive dependencies are ignored so aggressively, how about the reverse? dependencies present in lucene but not in maven? Tests should catch mosts of them i guess?

        Show
        selckin added a comment - Since transitive dependencies are ignored so aggressively, how about the reverse? dependencies present in lucene but not in maven? Tests should catch mosts of them i guess?
        Hide
        Chris Male added a comment -

        Yeah I think tests should be catching them. Do you have any examples?

        Show
        Chris Male added a comment - Yeah I think tests should be catching them. Do you have any examples?
        Hide
        Robert Muir added a comment -

        selckin: that problem is more than maven actually, e.g. things like SOLR-3736.

        Really you are right, if we have a dependency we should be testing that its working

        But its good to check the inverse, at least we know we are not depending on things we didnt know about, or the wrong versions: this
        only complements that.

        Show
        Robert Muir added a comment - selckin: that problem is more than maven actually, e.g. things like SOLR-3736 . Really you are right, if we have a dependency we should be testing that its working But its good to check the inverse, at least we know we are not depending on things we didnt know about, or the wrong versions: this only complements that.
        Hide
        Uwe Schindler added a comment -

        Looks fine now for Lucene! For consistency with the other maven macros, i would rename <macrodef name="validate-maven-dependencies"> to <macrodef name="m2-validate-dependencies">, then it is consistent with others. And maybe move it up to be next to each other, so not distributes over whole common-build.xml.

        Show
        Uwe Schindler added a comment - Looks fine now for Lucene! For consistency with the other maven macros, i would rename <macrodef name="validate-maven-dependencies"> to <macrodef name="m2-validate-dependencies">, then it is consistent with others. And maybe move it up to be next to each other, so not distributes over whole common-build.xml.
        Hide
        Uwe Schindler added a comment -

        One thing that is annoying:

        dist-maven-common:
        [artifact:dependencies] [INFO] snapshot org.apache.lucene:lucene-core:5.0-SNAPSHOT: checking for updates from sonatype.releases
        [artifact:dependencies] [INFO] snapshot org.apache.lucene:lucene-core:5.0-SNAPSHOT: checking for updates from apache.snapshots
        [artifact:dependencies] Downloading: org/apache/lucene/lucene-core/5.0-SNAPSHOT/lucene-core-5.0-20120903.170333-49.pom from reposito
        ry apache.snapshots at http://repository.apache.org/snapshots
        [artifact:dependencies] Downloading: org/apache/lucene/lucene-core/5.0-SNAPSHOT/lucene-core-5.0-20120903.170333-49.jar from reposito
        ry apache.snapshots at http://repository.apache.org/snapshots
        [artifact:dependencies] Transferring 2200K from apache.snapshots
             [echo] Checking Maven dependencies for C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr2\lucene\build\poms\lucene\test-frame
        work\pom.xml: com\carrotsearch\randomizedtesting\randomizedtesting-runner\2.0.0.rc5\randomizedtesting-runner-2.0.0.rc5.jar;junit\jun
        it\4.10\junit-4.10.jar;org\apache\lucene\lucene-core\5.0-SNAPSHOT\lucene-core-5.0-20120903.170333-49.jar
         [licenses] Scanned 2 JAR file(s) for licenses (in 0.01s.), 0 error(s).
        [artifact:install-provider] Installing provider: org.apache.maven.wagon:wagon-ssh:jar:1.0-beta-7:runtime
        [artifact:deploy] Deploying to file://C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr2\lucene\dist\maven
        [artifact:deploy] [INFO] Retrieving previous build number from ${m2.repository.id}
        [artifact:deploy] [INFO] repository metadata for: 'snapshot org.apache.lucene:lucene-test-framework:5.0-SNAPSHOT' could not be found
         on repository: ${m2.repository.id}, so will be created
        [artifact:deploy] Uploading: org/apache/lucene/lucene-test-framework/5.0-SNAPSHOT/lucene-test-framework-5.0-20120904.150305-1.jar to
         repository ${m2.repository.id} at file://C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr2\lucene\dist\maven
        [artifact:deploy] Transferring 5945K from ${m2.repository.id}
        [artifact:deploy] Uploaded 5945K
        [artifact:deploy] [INFO] Retrieving previous metadata from ${m2.repository.id}
        [artifact:deploy] [INFO] repository metadata for: 'artifact org.apache.lucene:lucene-test-framework' could not be found on repositor
        y: ${m2.repository.id}, so will be created
        [artifact:deploy] [INFO] Uploading repository metadata for: 'artifact org.apache.lucene:lucene-test-framework'
        [artifact:deploy] [INFO] Retrieving previous metadata from ${m2.repository.id}
        [artifact:deploy] [INFO] repository metadata for: 'snapshot org.apache.lucene:lucene-test-framework:5.0-SNAPSHOT' could not be found
         on repository: ${m2.repository.id}, so will be created
        [artifact:deploy] [INFO] Uploading repository metadata for: 'snapshot org.apache.lucene:lucene-test-framework:5.0-SNAPSHOT'
        [artifact:deploy] [INFO] Uploading project information for lucene-test-framework 5.0-20120904.150305-1
        [artifact:deploy] [INFO] Retrieving previous build number from ${m2.repository.id}
        [artifact:deploy] Uploading: org/apache/lucene/lucene-test-framework/5.0-SNAPSHOT/lucene-test-framework-5.0-20120904.150305-1-source
        s.jar to repository ${m2.repository.id} at file://C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr2\lucene\dist\maven
        [artifact:deploy] Transferring 5818K from ${m2.repository.id}
        [artifact:deploy] Uploaded 5818K
        [artifact:deploy] [INFO] Retrieving previous build number from ${m2.repository.id}
        [artifact:deploy] Uploading: org/apache/lucene/lucene-test-framework/5.0-SNAPSHOT/lucene-test-framework-5.0-20120904.150305-1-javado
        c.jar to repository ${m2.repository.id} at file://C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr2\lucene\dist\maven
        [artifact:deploy] Transferring 695K from ${m2.repository.id}
        [artifact:deploy] Uploaded 695K
        

        See first few lines. After nuking ~/.m2 it downloads all dependency JAR files from maven central, but it also downloads snapshot Lucene JARS, although it built them before. Can we supress this. Its not a problem for the license checker (because it excludes all those JARS), but i makes me nervous...

        Show
        Uwe Schindler added a comment - One thing that is annoying: dist-maven-common: [artifact:dependencies] [INFO] snapshot org.apache.lucene:lucene-core:5.0-SNAPSHOT: checking for updates from sonatype.releases [artifact:dependencies] [INFO] snapshot org.apache.lucene:lucene-core:5.0-SNAPSHOT: checking for updates from apache.snapshots [artifact:dependencies] Downloading: org/apache/lucene/lucene-core/5.0-SNAPSHOT/lucene-core-5.0-20120903.170333-49.pom from reposito ry apache.snapshots at http://repository.apache.org/snapshots [artifact:dependencies] Downloading: org/apache/lucene/lucene-core/5.0-SNAPSHOT/lucene-core-5.0-20120903.170333-49.jar from reposito ry apache.snapshots at http://repository.apache.org/snapshots [artifact:dependencies] Transferring 2200K from apache.snapshots [echo] Checking Maven dependencies for C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr2\lucene\build\poms\lucene\test-frame work\pom.xml: com\carrotsearch\randomizedtesting\randomizedtesting-runner\2.0.0.rc5\randomizedtesting-runner-2.0.0.rc5.jar;junit\jun it\4.10\junit-4.10.jar;org\apache\lucene\lucene-core\5.0-SNAPSHOT\lucene-core-5.0-20120903.170333-49.jar [licenses] Scanned 2 JAR file(s) for licenses (in 0.01s.), 0 error(s). [artifact:install-provider] Installing provider: org.apache.maven.wagon:wagon-ssh:jar:1.0-beta-7:runtime [artifact:deploy] Deploying to file://C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr2\lucene\dist\maven [artifact:deploy] [INFO] Retrieving previous build number from ${m2.repository.id} [artifact:deploy] [INFO] repository metadata for: 'snapshot org.apache.lucene:lucene-test-framework:5.0-SNAPSHOT' could not be found on repository: ${m2.repository.id}, so will be created [artifact:deploy] Uploading: org/apache/lucene/lucene-test-framework/5.0-SNAPSHOT/lucene-test-framework-5.0-20120904.150305-1.jar to repository ${m2.repository.id} at file://C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr2\lucene\dist\maven [artifact:deploy] Transferring 5945K from ${m2.repository.id} [artifact:deploy] Uploaded 5945K [artifact:deploy] [INFO] Retrieving previous metadata from ${m2.repository.id} [artifact:deploy] [INFO] repository metadata for: 'artifact org.apache.lucene:lucene-test-framework' could not be found on repositor y: ${m2.repository.id}, so will be created [artifact:deploy] [INFO] Uploading repository metadata for: 'artifact org.apache.lucene:lucene-test-framework' [artifact:deploy] [INFO] Retrieving previous metadata from ${m2.repository.id} [artifact:deploy] [INFO] repository metadata for: 'snapshot org.apache.lucene:lucene-test-framework:5.0-SNAPSHOT' could not be found on repository: ${m2.repository.id}, so will be created [artifact:deploy] [INFO] Uploading repository metadata for: 'snapshot org.apache.lucene:lucene-test-framework:5.0-SNAPSHOT' [artifact:deploy] [INFO] Uploading project information for lucene-test-framework 5.0-20120904.150305-1 [artifact:deploy] [INFO] Retrieving previous build number from ${m2.repository.id} [artifact:deploy] Uploading: org/apache/lucene/lucene-test-framework/5.0-SNAPSHOT/lucene-test-framework-5.0-20120904.150305-1-source s.jar to repository ${m2.repository.id} at file://C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr2\lucene\dist\maven [artifact:deploy] Transferring 5818K from ${m2.repository.id} [artifact:deploy] Uploaded 5818K [artifact:deploy] [INFO] Retrieving previous build number from ${m2.repository.id} [artifact:deploy] Uploading: org/apache/lucene/lucene-test-framework/5.0-SNAPSHOT/lucene-test-framework-5.0-20120904.150305-1-javado c.jar to repository ${m2.repository.id} at file://C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr2\lucene\dist\maven [artifact:deploy] Transferring 695K from ${m2.repository.id} [artifact:deploy] Uploaded 695K See first few lines. After nuking ~/.m2 it downloads all dependency JAR files from maven central, but it also downloads snapshot Lucene JARS, although it built them before. Can we supress this. Its not a problem for the license checker (because it excludes all those JARS), but i makes me nervous...
        Hide
        Robert Muir added a comment -

        I have no idea: thats a maven issue (its not really an issue with this license checking).

        Show
        Robert Muir added a comment - I have no idea: thats a maven issue (its not really an issue with this license checking).
        Hide
        Robert Muir added a comment -

        latest patch: semi-working for solr.

        there is currently a failure. Some stuff depends on commons-compress-1.3.jar and others on commons-compress-1.4.1.jar.

        We need to clear this up.

        Show
        Robert Muir added a comment - latest patch: semi-working for solr. there is currently a failure. Some stuff depends on commons-compress-1.3.jar and others on commons-compress-1.4.1.jar. We need to clear this up.
        Hide
        Robert Muir added a comment -

        I opened LUCENE-4356 to fix the commons-compress (there are 3 versions being used, which is crazy)

        Show
        Robert Muir added a comment - I opened LUCENE-4356 to fix the commons-compress (there are 3 versions being used, which is crazy)
        Hide
        Robert Muir added a comment -

        I also tripped LUCENE-4357. I think because i depend on compile-tools (not common.compile-tools, and icu redefines that).

        So the ICU tools got compiled too (which normally dont), and then forbidden APIs tripped.

        Show
        Robert Muir added a comment - I also tripped LUCENE-4357 . I think because i depend on compile-tools (not common.compile-tools, and icu redefines that). So the ICU tools got compiled too (which normally dont), and then forbidden APIs tripped.
        Hide
        Steve Rowe added a comment -

        After nuking ~/.m2 it downloads all dependency JAR files from maven central, but it also downloads snapshot Lucene JARS, although it built them before. Can we supress this. Its not a problem for the license checker (because it excludes all those JARS), but i makes me nervous...

        This happens because the Apache snapshot repo is listed in the poms, but it shouldn't be - we should never download Lucene/Solr snapshots as part of the build. I'll open an issue.

        Show
        Steve Rowe added a comment - After nuking ~/.m2 it downloads all dependency JAR files from maven central, but it also downloads snapshot Lucene JARS, although it built them before. Can we supress this. Its not a problem for the license checker (because it excludes all those JARS), but i makes me nervous... This happens because the Apache snapshot repo is listed in the poms, but it shouldn't be - we should never download Lucene/Solr snapshots as part of the build. I'll open an issue.
        Hide
        Steve Rowe added a comment -

        the Apache snapshot repo is listed in the poms, but it shouldn't be - we should never download Lucene/Solr snapshots as part of the build. I'll open an issue.

        LUCENE-4358

        Show
        Steve Rowe added a comment - the Apache snapshot repo is listed in the poms, but it shouldn't be - we should never download Lucene/Solr snapshots as part of the build. I'll open an issue. LUCENE-4358
        Hide
        Robert Muir added a comment -

        I will update the patch shortly merging in the changes to LUCENE-4356.

        Show
        Robert Muir added a comment - I will update the patch shortly merging in the changes to LUCENE-4356 .
        Hide
        Robert Muir added a comment -

        Updated patch to trunk. lucene/ and solr/ both pass now.

        So the only funky thing is the two servlet-api jars, which i disabled with a nocommit:

              <excludes>
                <rsel:or>
                  <rsel:name name="**/lucene-*.jar" handledirsep="true"/>
                  <rsel:name name="**/solr-*.jar" handledirsep="true"/>
                  <!-- nocommit -->
                  <rsel:name name="**/*servlet*.jar" handledirsep="true"/>
                </rsel:or>
              </excludes>
        

        I feel like this servlet-api crap is a bigger issue though and we shouldnt let it block this one.

        Show
        Robert Muir added a comment - Updated patch to trunk. lucene/ and solr/ both pass now. So the only funky thing is the two servlet-api jars, which i disabled with a nocommit: <excludes> <rsel:or> <rsel:name name="**/lucene-*.jar" handledirsep="true"/> <rsel:name name="**/solr-*.jar" handledirsep="true"/> <!-- nocommit --> <rsel:name name="**/*servlet*.jar" handledirsep="true"/> </rsel:or> </excludes> I feel like this servlet-api crap is a bigger issue though and we shouldnt let it block this one.
        Hide
        Robert Muir added a comment -

        The test part of this patch seems hung on some ancient maven bugs.

        I'll leave it be:

        but we need to apply the fixes to the poms themselves (the dev-tools part of this patch) because the
        maven dependencies are currently not correct.

        Show
        Robert Muir added a comment - The test part of this patch seems hung on some ancient maven bugs. I'll leave it be: but we need to apply the fixes to the poms themselves (the dev-tools part of this patch) because the maven dependencies are currently not correct.
        Hide
        Robert Muir added a comment -

        I committed this part to trunk. Hopefully i didnt break anything (I tested by generate-maven-artifacts plus using my checker).

        I'll merge to 4.x with the same test.

        If there are bugs just revert my commits, i just want the dependencies fixed first. the check itself is less important.

        Show
        Robert Muir added a comment - I committed this part to trunk. Hopefully i didnt break anything (I tested by generate-maven-artifacts plus using my checker). I'll merge to 4.x with the same test. If there are bugs just revert my commits, i just want the dependencies fixed first. the check itself is less important.
        Hide
        Uwe Schindler added a comment -

        OK!

        Maybe, we maybe only need a heavy commiting branch for the valiation task!

        Show
        Uwe Schindler added a comment - OK! Maybe, we maybe only need a heavy commiting branch for the valiation task!
        Hide
        Uwe Schindler added a comment -

        OK, can you add the half-baked check as a patch? Maybe Steven can help to also fix the validation one.

        Show
        Uwe Schindler added a comment - OK, can you add the half-baked check as a patch? Maybe Steven can help to also fix the validation one.
        Hide
        Robert Muir added a comment -

        Here is the patch excluding the dev-tools part (which i already committed).

        It applies clean to 4.x/trunk so i will just test it manually.

        I dont think its half-baked, its one and only goal is to test third-party dependencies. I could care less if it download the entire internet, or what it does with lucene-XXXX or solr-XXXX since it ignores those: its only to check third-party dependencies.

        Show
        Robert Muir added a comment - Here is the patch excluding the dev-tools part (which i already committed). It applies clean to 4.x/trunk so i will just test it manually. I dont think its half-baked, its one and only goal is to test third-party dependencies. I could care less if it download the entire internet, or what it does with lucene-XXXX or solr-XXXX since it ignores those: its only to check third-party dependencies.
        Hide
        Robert Muir added a comment -

        for 4.x we have to fix tika again. now maven depends on 1.1. But the upgrade to 1.2 was backported, so the checker currently fails (even with the patch here).
        Ill fix it to 1.2 and see if thats enough...

        Show
        Robert Muir added a comment - for 4.x we have to fix tika again. now maven depends on 1.1. But the upgrade to 1.2 was backported, so the checker currently fails (even with the patch here). Ill fix it to 1.2 and see if thats enough...
        Hide
        Robert Muir added a comment -

        ok same patch committed to 4.x too, only with the tika bump from 1.1 to 1.2 to match Jan's commit of r1380079.

        The latest patch up here passes for lucene and solr on both branches now.

        Show
        Robert Muir added a comment - ok same patch committed to 4.x too, only with the tika bump from 1.1 to 1.2 to match Jan's commit of r1380079. The latest patch up here passes for lucene and solr on both branches now.
        Hide
        Uwe Schindler added a comment - - edited

        Patch, that not only implements the validation, it also cleans up Maven-Tast/Traget dependnecies in ANT. I counted: All javadocs in Lucene modules were built 2 times, for analyis 3 times!

        This patch first runs generate-maven-artifacts as before (but faster) and can then run validate-maven-dependencies (which depends on generate-maven-artifacts). As this iterates all modules 2 times, you need lot's of perm-gen. In fact sometimes (if my repository is empty), not even generate-maven-artifacts passes without more permgen (Windows 7 64).

        The patch also adds the undefined ant property m2.repository.id (defaults to "local").

        Show
        Uwe Schindler added a comment - - edited Patch, that not only implements the validation, it also cleans up Maven-Tast/Traget dependnecies in ANT. I counted: All javadocs in Lucene modules were built 2 times, for analyis 3 times! This patch first runs generate-maven-artifacts as before (but faster) and can then run validate-maven-dependencies (which depends on generate-maven-artifacts). As this iterates all modules 2 times, you need lot's of perm-gen. In fact sometimes (if my repository is empty), not even generate-maven-artifacts passes without more permgen (Windows 7 64). The patch also adds the undefined ant property m2.repository.id (defaults to "local").
        Hide
        Robert Muir added a comment -

        Hi Uwe, thanks!

        I think we can call the internal task -validate-maven-dependencies and remove its depends,
        since the top-level one already depends on generate-maven-artifacts. so this work is already done...
        this should also make my permgen happier.

        Show
        Robert Muir added a comment - Hi Uwe, thanks! I think we can call the internal task -validate-maven-dependencies and remove its depends, since the top-level one already depends on generate-maven-artifacts. so this work is already done... this should also make my permgen happier.
        Hide
        Uwe Schindler added a comment -

        Fix for solr (missed to rename some targets)

        Show
        Uwe Schindler added a comment - Fix for solr (missed to rename some targets)
        Hide
        Robert Muir added a comment -

        not even generate-maven-artifacts passes without more permgen (Windows 7 64).

        Thats because you were a bit over-eager here:

        -  <target name="generate-maven-artifacts"
        -          depends="install-maven-tasks, filter-pom-templates, package, javadocs">
        +  <target name="generate-maven-artifacts">
        

        instead, depend on install-maven-tasks here (it seems redundant, its not).
        its then loaded once, and the property is passed in the uptodate.and.compiled.properties and
        you will see in the log that its task is only installed once rather than over-and-over

        I fixed this a while ago and i fixed it locally in your patch (along with similar stuff) and it clears this up fine.
        I'll take care of it (with a note not to remove the depends), I just wanted to explain!

        Show
        Robert Muir added a comment - not even generate-maven-artifacts passes without more permgen (Windows 7 64). Thats because you were a bit over-eager here: - <target name="generate-maven-artifacts" - depends="install-maven-tasks, filter-pom-templates, package, javadocs"> + <target name="generate-maven-artifacts"> instead, depend on install-maven-tasks here (it seems redundant, its not). its then loaded once, and the property is passed in the uptodate.and.compiled.properties and you will see in the log that its task is only installed once rather than over-and-over I fixed this a while ago and i fixed it locally in your patch (along with similar stuff) and it clears this up fine. I'll take care of it (with a note not to remove the depends), I just wanted to explain!
        Hide
        Robert Muir added a comment -

        Uwe's patch: but without permgen problems

        Show
        Robert Muir added a comment - Uwe's patch: but without permgen problems
        Hide
        Robert Muir added a comment -

        synced up patch to trunk.

        One ugly thing in solr is that to validate the dependencies, we need to ensure the lucene ones are installed.

        for now this is ensured, since its only a validation/test task for jenkins anyway and better to be safe i think?

        Show
        Robert Muir added a comment - synced up patch to trunk. One ugly thing in solr is that to validate the dependencies, we need to ensure the lucene ones are installed. for now this is ensured, since its only a validation/test task for jenkins anyway and better to be safe i think?
        Hide
        Uwe Schindler added a comment -

        Robert: I accidentally deleted your latest patch (I got confused with the order of patches). Can you reattach?

        Show
        Uwe Schindler added a comment - Robert: I accidentally deleted your latest patch (I got confused with the order of patches). Can you reattach?
        Hide
        Robert Muir added a comment -

        here is the latest patch

        Show
        Robert Muir added a comment - here is the latest patch
        Hide
        Robert Muir added a comment -

        merged patch up to trunk changes.

        Show
        Robert Muir added a comment - merged patch up to trunk changes.
        Hide
        Robert Muir added a comment -

        I committed this: but its not yet linked into any jenkins job, until this randomizedtesting jar can actually be downloaded from maven reliably.

        Show
        Robert Muir added a comment - I committed this: but its not yet linked into any jenkins job, until this randomizedtesting jar can actually be downloaded from maven reliably.

          People

          • Assignee:
            Unassigned
            Reporter:
            Robert Muir
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Development