Lucene - Core
  1. Lucene - Core
  2. LUCENE-5217

disable transitive dependencies in maven config

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.7, 6.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Our ivy configuration does this: each dependency is specified and so we know what will happen. Unfortunately the maven setup is not configured the same way.

      Instead the maven setup is configured to download the internet: and it excludes certain things specifically.

      This is really hard to configure and maintain: we added a 'validate-maven-dependencies' that tries to fail on any extra jars, but all it really does is run a license check after maven "runs". It wouldnt find unnecessary dependencies being dragged in if something else in lucene was using them and thus they had a license file.

      Since maven supports wildcard exclusions: MNG-3832, we can disable this transitive shit completely.

      We should do this, so its configuration is the exact parallel of ivy.

      1. LUCENE-5217.patch
        153 kB
        Steve Rowe
      2. LUCENE-5217.patch
        153 kB
        Steve Rowe
      3. LUCENE-5217.patch
        152 kB
        Steve Rowe
      4. LUCENE-5217.patch
        132 kB
        Steve Rowe

        Issue Links

          Activity

          Hide
          Robert Muir added a comment -

          This is also described here: http://www.smartjava.org/content/maven-and-wildcard-exclusions

          I think it just means we have to require a minimum of maven 3 instead of also supporting 2. Since this has been out for 3 years (in fact older than the ant 1.8.2 that we require), I don't see this as a significant imposition on anyone?

          Show
          Robert Muir added a comment - This is also described here: http://www.smartjava.org/content/maven-and-wildcard-exclusions I think it just means we have to require a minimum of maven 3 instead of also supporting 2. Since this has been out for 3 years (in fact older than the ant 1.8.2 that we require), I don't see this as a significant imposition on anyone?
          Hide
          Steve Rowe added a comment -

          This is really hard to configure and maintain

          I agree.

          maven supports wildcard exclusions: MNG-3832

          I did not know that.

          I think it just means we have to require a minimum of maven 3 instead of also supporting 2. Since this has been out for 3 years (in fact older than the ant 1.8.2 that we require), I don't see this as a significant imposition on anyone?

          +1, though this will be a viral change, unlike the Ant upgrade: for Ant, we only forced Lucene/Solr source users to upgrade, but for Maven, everybody who depends on binary Lucene/Solr artifacts will have to upgrade their own projects to Maven 3 - I think. I'll do some testing to confirm.

          Show
          Steve Rowe added a comment - This is really hard to configure and maintain I agree. maven supports wildcard exclusions: MNG-3832 I did not know that. I think it just means we have to require a minimum of maven 3 instead of also supporting 2. Since this has been out for 3 years (in fact older than the ant 1.8.2 that we require), I don't see this as a significant imposition on anyone? +1, though this will be a viral change, unlike the Ant upgrade: for Ant, we only forced Lucene/Solr source users to upgrade, but for Maven, everybody who depends on binary Lucene/Solr artifacts will have to upgrade their own projects to Maven 3 - I think. I'll do some testing to confirm.
          Hide
          Robert Muir added a comment -

          I wont comment on "viral change"

          But I think this is a totally fair thing to do for 5.0, since its a new major release.

          Show
          Robert Muir added a comment - I wont comment on "viral change" But I think this is a totally fair thing to do for 5.0, since its a new major release.
          Hide
          Steve Rowe added a comment - - edited

          I can't get the wildcard-exclusion-disables-transitive-dependencies thing to work - I tried with Maven 3.0.3, 3.0.5 and 3.1.0 (on OS X 10.8.4), all behave the same.

          I tested a single Solr module and looked at a single dependency's transitive dependency: solrj module, zookeeper dependency, and zookeeper's transitive dependency netty.

          Here's the zookeeper dependency config from solrj's POM:

          <dependency>
            <groupId>org.apache.zookeeper</groupId>
            <artifactId>zookeeper</artifactId>
            <exclusions>
              <exclusion>
                <groupId>javax.jms</groupId>
                <artifactId>jms</artifactId>
              </exclusion>
              <exclusion>
                <groupId>com.sun.jmx</groupId>
                <artifactId>jmxri</artifactId>
              </exclusion>
              <exclusion>
                <groupId>com.sun.jdmk</groupId>
                <artifactId>jmxtools</artifactId>
              </exclusion>
              <exclusion>
                <groupId>log4j</groupId>
                <artifactId>log4j</artifactId>
              </exclusion>
              <exclusion>
                <groupId>org.jboss.netty</groupId>
                <artifactId>netty</artifactId>
              </exclusion>
              <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-log4j12</artifactId>
              </exclusion>
              <exclusion>
                <groupId>jline</groupId>
                <artifactId>jline</artifactId>
              </exclusion>
            </exclusions>
          </dependency>
          

          Without making any changes, here's the output from running mvn -X -DskipTests install under maven-build/solr/solrj/src/java/:

          [INFO] --- maven-remote-resources-plugin:1.4:process (default) @ solr-solrj ---
          [...]
          [DEBUG] org.apache.solr:solr-solrj:jar:5.0-SNAPSHOT (selected for null)
          [DEBUG]   org.apache.zookeeper:zookeeper:jar:3.4.5:compile (selected for compile)
          [DEBUG]     org.slf4j:slf4j-api:jar:1.6.1:compile (applying version: 1.6.6)
          [DEBUG]     org.slf4j:slf4j-api:jar:1.6.6:compile (selected for compile)
          [DEBUG]   commons-io:commons-io:jar:2.1:compile (selected for compile)
          [DEBUG]   org.noggit:noggit:jar:0.5:compile (selected for compile)
          [DEBUG]   org.apache.httpcomponents:httpclient:jar:4.2.3:compile (selected for compile)
          [DEBUG]     org.apache.httpcomponents:httpcore:jar:4.2.2:compile (selected for compile)
          [DEBUG]     commons-logging:commons-logging:jar:1.1.1:compile (selected for compile)
          [DEBUG]     commons-codec:commons-codec:jar:1.6:compile (applying version: 1.7)
          [DEBUG]     commons-codec:commons-codec:jar:1.7:compile (selected for compile)
          [DEBUG]   org.apache.httpcomponents:httpmime:jar:4.2.3:compile (selected for compile)
          [DEBUG]   org.codehaus.woodstox:wstx-asl:jar:3.2.7:runtime (selected for runtime)
          [DEBUG]     stax:stax-api:jar:1.0.1:runtime (selected for runtime)
          [DEBUG]   org.slf4j:jcl-over-slf4j:jar:1.6.6:compile (selected for compile)
          [DEBUG]   org.slf4j:jul-to-slf4j:jar:1.6.6:compile (selected for compile)
          [DEBUG]   org.slf4j:slf4j-api:jar:1.6.6:compile (selected for compile)
          [DEBUG]   org.slf4j:slf4j-log4j12:jar:1.6.6:compile (selected for compile)
          [DEBUG]     log4j:log4j:jar:1.2.17:compile (applying version: 1.2.16)
          [DEBUG]     log4j:log4j:jar:1.2.16:compile (selected for compile)
          [DEBUG]   log4j:log4j:jar:1.2.16:compile (selected for compile)
          [DEBUG]   junit:junit:jar:4.10:test (selected for test)
          

          You can see above that zookeeper's only transitive dependency is slf4j-api.

          I get the same output (and behavior) when I remove netty from my local Maven repository:

          rm -rf ~/.m2/repository/org/jboss/netty
          

          Even when netty isn't in my local repository, Maven doesn't try to download it.

          However, when I change the solrj POM to use the wildcard exclusions:

          <dependency>
            <groupId>org.apache.zookeeper</groupId>
            <artifactId>zookeeper</artifactId>
            <exclusions>
              <exclusion>
                <groupId>*</groupId>
                <artifactId>*</artifactId>
              </exclusion>
            </exclusions>
          </dependency>
          

          I get the following output from running mvn -X -DskipTests install under maven-build/solr/solrj/src/java/:

          [INFO] --- maven-remote-resources-plugin:1.4:process (default) @ solr-solrj ---
          [...]
          [DEBUG] org.apache.solr:solr-solrj:jar:5.0-SNAPSHOT (selected for null)
          [DEBUG]   org.apache.zookeeper:zookeeper:jar:3.4.5:compile (selected for compile)
          [DEBUG]     org.slf4j:slf4j-api:jar:1.6.1:compile (applying version: 1.6.6)
          [DEBUG]     org.slf4j:slf4j-api:jar:1.6.6:compile (selected for compile)
          [DEBUG]     org.slf4j:slf4j-log4j12:jar:1.6.1:compile (applying version: 1.6.6)
          [DEBUG]     org.slf4j:slf4j-log4j12:jar:1.6.6:compile (selected for compile)
          [DEBUG]       log4j:log4j:jar:1.2.17:compile (applying version: 1.2.16)
          [DEBUG]       log4j:log4j:jar:1.2.16:compile (selected for compile)
          [DEBUG]     log4j:log4j:jar:1.2.15:compile (applying version: 1.2.16)
          [DEBUG]     log4j:log4j:jar:1.2.16:compile (selected for compile)
          [DEBUG]     jline:jline:jar:0.9.94:compile (applying version: 0.9.1)
          [DEBUG]     jline:jline:jar:0.9.1:compile (selected for compile)
          [DEBUG] Using connector WagonRepositoryConnector with priority 0.0 for http://maven.restlet.org
          Downloading: http://maven.restlet.org/org/jboss/netty/netty/3.2.2.Final/netty-3.2.2.Final.pom
          [DEBUG] Writing tracking file /Users/sarowe/.m2/repository/org/jboss/netty/netty/3.2.2.Final/netty-3.2.2.Final.pom.lastUpdated
          [DEBUG] Using connector WagonRepositoryConnector with priority 0.0 for http://repo.maven.apache.org/maven2
          Downloading: http://repo.maven.apache.org/maven2/org/jboss/netty/netty/3.2.2.Final/netty-3.2.2.Final.pom
          Downloaded: http://repo.maven.apache.org/maven2/org/jboss/netty/netty/3.2.2.Final/netty-3.2.2.Final.pom (24 KB at 44.4 KB/sec)
          [DEBUG] Writing tracking file /Users/sarowe/.m2/repository/org/jboss/netty/netty/3.2.2.Final/netty-3.2.2.Final.pom.lastUpdated
          [DEBUG] Writing tracking file /Users/sarowe/.m2/repository/org/jboss/netty/netty/3.2.2.Final/_remote.repositories
          [DEBUG]     org.jboss.netty:netty:jar:3.2.2.Final:compile (selected for compile)
          [DEBUG]   commons-io:commons-io:jar:2.1:compile (selected for compile)
          [DEBUG]   org.noggit:noggit:jar:0.5:compile (selected for compile)
          [DEBUG]   org.apache.httpcomponents:httpclient:jar:4.2.3:compile (selected for compile)
          [DEBUG]     org.apache.httpcomponents:httpcore:jar:4.2.2:compile (selected for compile)
          [DEBUG]     commons-logging:commons-logging:jar:1.1.1:compile (selected for compile)
          [DEBUG]     commons-codec:commons-codec:jar:1.6:compile (applying version: 1.7)
          [DEBUG]     commons-codec:commons-codec:jar:1.7:compile (selected for compile)
          [DEBUG]   org.apache.httpcomponents:httpmime:jar:4.2.3:compile (selected for compile)
          [DEBUG]   org.codehaus.woodstox:wstx-asl:jar:3.2.7:runtime (selected for runtime)
          [DEBUG]     stax:stax-api:jar:1.0.1:runtime (selected for runtime)
          [DEBUG]   org.slf4j:jcl-over-slf4j:jar:1.6.6:compile (selected for compile)
          [DEBUG]   org.slf4j:jul-to-slf4j:jar:1.6.6:compile (selected for compile)
          [DEBUG]   org.slf4j:slf4j-api:jar:1.6.6:compile (selected for compile)
          [DEBUG]   org.slf4j:slf4j-log4j12:jar:1.6.6:compile (selected for compile)
          [DEBUG]     log4j:log4j:jar:1.2.17:compile (applying version: 1.2.16)
          [DEBUG]   log4j:log4j:jar:1.2.16:compile (selected for compile)
          [DEBUG]   junit:junit:jar:4.10:test (selected for test)
          [DEBUG] Using connector WagonRepositoryConnector with priority 0.0 for http://maven.restlet.org
          Downloading: http://maven.restlet.org/org/jboss/netty/netty/3.2.2.Final/netty-3.2.2.Final.jar
          [DEBUG] Writing tracking file /Users/sarowe/.m2/repository/org/jboss/netty/netty/3.2.2.Final/netty-3.2.2.Final.jar.lastUpdated
          [DEBUG] Using connector WagonRepositoryConnector with priority 0.0 for http://repo.maven.apache.org/maven2
          Downloading: http://repo.maven.apache.org/maven2/org/jboss/netty/netty/3.2.2.Final/netty-3.2.2.Final.jar
          Downloaded: http://repo.maven.apache.org/maven2/org/jboss/netty/netty/3.2.2.Final/netty-3.2.2.Final.jar (768 KB at 1839.7 KB/sec)
          [DEBUG] Writing tracking file /Users/sarowe/.m2/repository/org/jboss/netty/netty/3.2.2.Final/netty-3.2.2.Final.jar.lastUpdated
          [DEBUG] Writing tracking file /Users/sarowe/.m2/repository/org/jboss/netty/netty/3.2.2.Final/_remote.repositories
          

          You can see above that Maven is effectively ignoring the wildcard exception - the zookeeper dependency now has lots of transitive dependencies, and netty, the one I had removed from my local repo, gets downloaded.

          So it looks like this supposed workaround doesn't work at all?

          Confusingly, mvn dependency:tree seems to respect the transitive dependency disabling intent of wildcard exclusions - zookeeper has no transitive deps:

          [INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ solr-solrj ---
          [INFO] org.apache.solr:solr-solrj:jar:5.0-SNAPSHOT
          [INFO] +- org.apache.zookeeper:zookeeper:jar:3.4.5:compile
          [INFO] +- commons-io:commons-io:jar:2.1:compile
          [INFO] +- org.noggit:noggit:jar:0.5:compile
          [INFO] +- org.apache.httpcomponents:httpclient:jar:4.2.3:compile
          [INFO] +- org.apache.httpcomponents:httpmime:jar:4.2.3:compile
          [INFO] |  \- org.apache.httpcomponents:httpcore:jar:4.2.2:compile
          [INFO] +- org.codehaus.woodstox:wstx-asl:jar:3.2.7:runtime
          [INFO] |  \- stax:stax-api:jar:1.0.1:runtime
          [INFO] +- org.slf4j:jcl-over-slf4j:jar:1.6.6:compile
          [INFO] +- org.slf4j:jul-to-slf4j:jar:1.6.6:compile
          [INFO] +- org.slf4j:slf4j-api:jar:1.6.6:compile
          [INFO] +- org.slf4j:slf4j-log4j12:jar:1.6.6:compile
          [INFO] +- log4j:log4j:jar:1.2.16:compile
          [INFO] \- junit:junit:jar:4.10:test
          
          Show
          Steve Rowe added a comment - - edited I can't get the wildcard-exclusion-disables-transitive-dependencies thing to work - I tried with Maven 3.0.3, 3.0.5 and 3.1.0 (on OS X 10.8.4), all behave the same. I tested a single Solr module and looked at a single dependency's transitive dependency: solrj module, zookeeper dependency, and zookeeper's transitive dependency netty. Here's the zookeeper dependency config from solrj's POM: <dependency> <groupId>org.apache.zookeeper</groupId> <artifactId>zookeeper</artifactId> <exclusions> <exclusion> <groupId>javax.jms</groupId> <artifactId>jms</artifactId> </exclusion> <exclusion> <groupId>com.sun.jmx</groupId> <artifactId>jmxri</artifactId> </exclusion> <exclusion> <groupId>com.sun.jdmk</groupId> <artifactId>jmxtools</artifactId> </exclusion> <exclusion> <groupId>log4j</groupId> <artifactId>log4j</artifactId> </exclusion> <exclusion> <groupId>org.jboss.netty</groupId> <artifactId>netty</artifactId> </exclusion> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> </exclusion> <exclusion> <groupId>jline</groupId> <artifactId>jline</artifactId> </exclusion> </exclusions> </dependency> Without making any changes, here's the output from running mvn -X -DskipTests install under maven-build/solr/solrj/src/java/ : [INFO] --- maven-remote-resources-plugin:1.4:process (default) @ solr-solrj --- [...] [DEBUG] org.apache.solr:solr-solrj:jar:5.0-SNAPSHOT (selected for null) [DEBUG] org.apache.zookeeper:zookeeper:jar:3.4.5:compile (selected for compile) [DEBUG] org.slf4j:slf4j-api:jar:1.6.1:compile (applying version: 1.6.6) [DEBUG] org.slf4j:slf4j-api:jar:1.6.6:compile (selected for compile) [DEBUG] commons-io:commons-io:jar:2.1:compile (selected for compile) [DEBUG] org.noggit:noggit:jar:0.5:compile (selected for compile) [DEBUG] org.apache.httpcomponents:httpclient:jar:4.2.3:compile (selected for compile) [DEBUG] org.apache.httpcomponents:httpcore:jar:4.2.2:compile (selected for compile) [DEBUG] commons-logging:commons-logging:jar:1.1.1:compile (selected for compile) [DEBUG] commons-codec:commons-codec:jar:1.6:compile (applying version: 1.7) [DEBUG] commons-codec:commons-codec:jar:1.7:compile (selected for compile) [DEBUG] org.apache.httpcomponents:httpmime:jar:4.2.3:compile (selected for compile) [DEBUG] org.codehaus.woodstox:wstx-asl:jar:3.2.7:runtime (selected for runtime) [DEBUG] stax:stax-api:jar:1.0.1:runtime (selected for runtime) [DEBUG] org.slf4j:jcl-over-slf4j:jar:1.6.6:compile (selected for compile) [DEBUG] org.slf4j:jul-to-slf4j:jar:1.6.6:compile (selected for compile) [DEBUG] org.slf4j:slf4j-api:jar:1.6.6:compile (selected for compile) [DEBUG] org.slf4j:slf4j-log4j12:jar:1.6.6:compile (selected for compile) [DEBUG] log4j:log4j:jar:1.2.17:compile (applying version: 1.2.16) [DEBUG] log4j:log4j:jar:1.2.16:compile (selected for compile) [DEBUG] log4j:log4j:jar:1.2.16:compile (selected for compile) [DEBUG] junit:junit:jar:4.10:test (selected for test) You can see above that zookeeper's only transitive dependency is slf4j-api. I get the same output (and behavior) when I remove netty from my local Maven repository: rm -rf ~/.m2/repository/org/jboss/netty Even when netty isn't in my local repository, Maven doesn't try to download it. However, when I change the solrj POM to use the wildcard exclusions: <dependency> <groupId>org.apache.zookeeper</groupId> <artifactId>zookeeper</artifactId> <exclusions> <exclusion> <groupId>*</groupId> <artifactId>*</artifactId> </exclusion> </exclusions> </dependency> I get the following output from running mvn -X -DskipTests install under maven-build/solr/solrj/src/java/ : [INFO] --- maven-remote-resources-plugin:1.4:process (default) @ solr-solrj --- [...] [DEBUG] org.apache.solr:solr-solrj:jar:5.0-SNAPSHOT (selected for null) [DEBUG] org.apache.zookeeper:zookeeper:jar:3.4.5:compile (selected for compile) [DEBUG] org.slf4j:slf4j-api:jar:1.6.1:compile (applying version: 1.6.6) [DEBUG] org.slf4j:slf4j-api:jar:1.6.6:compile (selected for compile) [DEBUG] org.slf4j:slf4j-log4j12:jar:1.6.1:compile (applying version: 1.6.6) [DEBUG] org.slf4j:slf4j-log4j12:jar:1.6.6:compile (selected for compile) [DEBUG] log4j:log4j:jar:1.2.17:compile (applying version: 1.2.16) [DEBUG] log4j:log4j:jar:1.2.16:compile (selected for compile) [DEBUG] log4j:log4j:jar:1.2.15:compile (applying version: 1.2.16) [DEBUG] log4j:log4j:jar:1.2.16:compile (selected for compile) [DEBUG] jline:jline:jar:0.9.94:compile (applying version: 0.9.1) [DEBUG] jline:jline:jar:0.9.1:compile (selected for compile) [DEBUG] Using connector WagonRepositoryConnector with priority 0.0 for http://maven.restlet.org Downloading: http://maven.restlet.org/org/jboss/netty/netty/3.2.2.Final/netty-3.2.2.Final.pom [DEBUG] Writing tracking file /Users/sarowe/.m2/repository/org/jboss/netty/netty/3.2.2.Final/netty-3.2.2.Final.pom.lastUpdated [DEBUG] Using connector WagonRepositoryConnector with priority 0.0 for http://repo.maven.apache.org/maven2 Downloading: http://repo.maven.apache.org/maven2/org/jboss/netty/netty/3.2.2.Final/netty-3.2.2.Final.pom Downloaded: http://repo.maven.apache.org/maven2/org/jboss/netty/netty/3.2.2.Final/netty-3.2.2.Final.pom (24 KB at 44.4 KB/sec) [DEBUG] Writing tracking file /Users/sarowe/.m2/repository/org/jboss/netty/netty/3.2.2.Final/netty-3.2.2.Final.pom.lastUpdated [DEBUG] Writing tracking file /Users/sarowe/.m2/repository/org/jboss/netty/netty/3.2.2.Final/_remote.repositories [DEBUG] org.jboss.netty:netty:jar:3.2.2.Final:compile (selected for compile) [DEBUG] commons-io:commons-io:jar:2.1:compile (selected for compile) [DEBUG] org.noggit:noggit:jar:0.5:compile (selected for compile) [DEBUG] org.apache.httpcomponents:httpclient:jar:4.2.3:compile (selected for compile) [DEBUG] org.apache.httpcomponents:httpcore:jar:4.2.2:compile (selected for compile) [DEBUG] commons-logging:commons-logging:jar:1.1.1:compile (selected for compile) [DEBUG] commons-codec:commons-codec:jar:1.6:compile (applying version: 1.7) [DEBUG] commons-codec:commons-codec:jar:1.7:compile (selected for compile) [DEBUG] org.apache.httpcomponents:httpmime:jar:4.2.3:compile (selected for compile) [DEBUG] org.codehaus.woodstox:wstx-asl:jar:3.2.7:runtime (selected for runtime) [DEBUG] stax:stax-api:jar:1.0.1:runtime (selected for runtime) [DEBUG] org.slf4j:jcl-over-slf4j:jar:1.6.6:compile (selected for compile) [DEBUG] org.slf4j:jul-to-slf4j:jar:1.6.6:compile (selected for compile) [DEBUG] org.slf4j:slf4j-api:jar:1.6.6:compile (selected for compile) [DEBUG] org.slf4j:slf4j-log4j12:jar:1.6.6:compile (selected for compile) [DEBUG] log4j:log4j:jar:1.2.17:compile (applying version: 1.2.16) [DEBUG] log4j:log4j:jar:1.2.16:compile (selected for compile) [DEBUG] junit:junit:jar:4.10:test (selected for test) [DEBUG] Using connector WagonRepositoryConnector with priority 0.0 for http://maven.restlet.org Downloading: http://maven.restlet.org/org/jboss/netty/netty/3.2.2.Final/netty-3.2.2.Final.jar [DEBUG] Writing tracking file /Users/sarowe/.m2/repository/org/jboss/netty/netty/3.2.2.Final/netty-3.2.2.Final.jar.lastUpdated [DEBUG] Using connector WagonRepositoryConnector with priority 0.0 for http://repo.maven.apache.org/maven2 Downloading: http://repo.maven.apache.org/maven2/org/jboss/netty/netty/3.2.2.Final/netty-3.2.2.Final.jar Downloaded: http://repo.maven.apache.org/maven2/org/jboss/netty/netty/3.2.2.Final/netty-3.2.2.Final.jar (768 KB at 1839.7 KB/sec) [DEBUG] Writing tracking file /Users/sarowe/.m2/repository/org/jboss/netty/netty/3.2.2.Final/netty-3.2.2.Final.jar.lastUpdated [DEBUG] Writing tracking file /Users/sarowe/.m2/repository/org/jboss/netty/netty/3.2.2.Final/_remote.repositories You can see above that Maven is effectively ignoring the wildcard exception - the zookeeper dependency now has lots of transitive dependencies, and netty, the one I had removed from my local repo, gets downloaded. So it looks like this supposed workaround doesn't work at all? Confusingly, mvn dependency:tree seems to respect the transitive dependency disabling intent of wildcard exclusions - zookeeper has no transitive deps: [INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ solr-solrj --- [INFO] org.apache.solr:solr-solrj:jar:5.0-SNAPSHOT [INFO] +- org.apache.zookeeper:zookeeper:jar:3.4.5:compile [INFO] +- commons-io:commons-io:jar:2.1:compile [INFO] +- org.noggit:noggit:jar:0.5:compile [INFO] +- org.apache.httpcomponents:httpclient:jar:4.2.3:compile [INFO] +- org.apache.httpcomponents:httpmime:jar:4.2.3:compile [INFO] | \- org.apache.httpcomponents:httpcore:jar:4.2.2:compile [INFO] +- org.codehaus.woodstox:wstx-asl:jar:3.2.7:runtime [INFO] | \- stax:stax-api:jar:1.0.1:runtime [INFO] +- org.slf4j:jcl-over-slf4j:jar:1.6.6:compile [INFO] +- org.slf4j:jul-to-slf4j:jar:1.6.6:compile [INFO] +- org.slf4j:slf4j-api:jar:1.6.6:compile [INFO] +- org.slf4j:slf4j-log4j12:jar:1.6.6:compile [INFO] +- log4j:log4j:jar:1.2.16:compile [INFO] \- junit:junit:jar:4.10:test
          Hide
          David Smiley added a comment -

          I have a proposal that may address the fundamental concern – and the concern is that modifications to our maven pom's could bring inadvertent/unintentional transitive dependencies.

          What if we check-in to source control the output of "mvn dependency:tree" – at least the essential part of the output. Then, as part of the nightly Continuous-Integration build, an ant task vets that the checked-in dependency tree matches the current output. Cool?

          Show
          David Smiley added a comment - I have a proposal that may address the fundamental concern – and the concern is that modifications to our maven pom's could bring inadvertent/unintentional transitive dependencies. What if we check-in to source control the output of "mvn dependency:tree" – at least the essential part of the output. Then, as part of the nightly Continuous-Integration build, an ant task vets that the checked-in dependency tree matches the current output. Cool?
          Hide
          Robert Muir added a comment -

          more/faster/simpler ways to test that the maven config == ivy config would help.

          But my original concern is actually that i greatly prefer a "whitelist" like ivy, where you just say transitive=false and declare exactly what you want, and we know from the ivy.xml exactly what is being downloaded just by looking at it.

          With maven it seems it wants you to make a "blacklist" of what you dont want, and you don't have any idea what is really being downloaded by looking at the configuration.

          Show
          Robert Muir added a comment - more/faster/simpler ways to test that the maven config == ivy config would help. But my original concern is actually that i greatly prefer a "whitelist" like ivy, where you just say transitive=false and declare exactly what you want, and we know from the ivy.xml exactly what is being downloaded just by looking at it. With maven it seems it wants you to make a "blacklist" of what you dont want, and you don't have any idea what is really being downloaded by looking at the configuration.
          Hide
          Steve Rowe added a comment -

          I tried <ivy:makepom> on the solrj module - I had hoped that it would take the <dependency [...] transitive="false"/> directives in our ivy.xml files and populate the appropriate exclusions in the output POM, but it doesn't - here's the zookeeper dependency from its output:

          <dependency>
             <groupId>org.apache.zookeeper</groupId>
             <artifactId>zookeeper</artifactId>
             <version>3.4.5</version>
             <optional>true</optional>
          </dependency>
          
          Show
          Steve Rowe added a comment - I tried <ivy:makepom> on the solrj module - I had hoped that it would take the <dependency [...] transitive="false" /> directives in our ivy.xml files and populate the appropriate exclusions in the output POM, but it doesn't - here's the zookeeper dependency from its output: <dependency> <groupId>org.apache.zookeeper</groupId> <artifactId>zookeeper</artifactId> <version>3.4.5</version> <optional>true</optional> </dependency>
          Hide
          Steve Rowe added a comment -

          Patch that templatizes dependencies in all POM templates, and fills them out using a new Ant task.

          In the <dependencyManagement> section in the grandparent POM produced using this patch, all dependencies' versions are specified, as are exclusions for all transitive compile-scope dependencies.

          So this patch directly makes the Maven setup the same as the Ant setup: all transitive dependencies are excluded, and all dependencies and their versions are pulled from the Ant/Ivy config.

          This is a work in progress: I've only tested the get-maven-poms task so far, and I haven't yet run the Maven build or tried to generate-maven-artifacts, so it's not ready yet. I think it's most of the way there though.

          The strategy is to put all modules' classpath-s and test.classpath-s into a single properties file using vanilla Ant, and then feed that into a new custom Ant task that also reads all the ivy.xml files, the centralized ivy versions file, and the external dependencies' ivy.xml files in the Ivy cache, then output a filters file that is used when copy'ing the POM templates to maven-build/, so that the appropriate spots are filled in with the right stuff.

          It's kludgy, in that properties files are used to communicate some information, rather than directly, but I can't see a way to do it all in memory.

          Don't be scared by the size of the patch: the majority is just removing the hard-coded dependencies in the POM templates.

          Show
          Steve Rowe added a comment - Patch that templatizes dependencies in all POM templates, and fills them out using a new Ant task. In the <dependencyManagement> section in the grandparent POM produced using this patch, all dependencies' versions are specified, as are exclusions for all transitive compile-scope dependencies. So this patch directly makes the Maven setup the same as the Ant setup: all transitive dependencies are excluded, and all dependencies and their versions are pulled from the Ant/Ivy config. This is a work in progress: I've only tested the get-maven-poms task so far, and I haven't yet run the Maven build or tried to generate-maven-artifacts , so it's not ready yet. I think it's most of the way there though. The strategy is to put all modules' classpath -s and test.classpath -s into a single properties file using vanilla Ant, and then feed that into a new custom Ant task that also reads all the ivy.xml files, the centralized ivy versions file, and the external dependencies' ivy.xml files in the Ivy cache, then output a filters file that is used when copy'ing the POM templates to maven-build/ , so that the appropriate spots are filled in with the right stuff. It's kludgy, in that properties files are used to communicate some information, rather than directly, but I can't see a way to do it all in memory. Don't be scared by the size of the patch: the majority is just removing the hard-coded dependencies in the POM templates.
          Hide
          Steve Rowe added a comment -

          New patch, almost there.

          Changes:

          • ivy.xml files and ivy-<version>.xml files from the Ivy cache are now parsed with DOM+XPath instead of SAX.
          • All tests pass under the maven build using the filtered POMs.
          • More javadocs added.
          • The filtered grandparent POM is now over 5,000 lines long (previously about 900 lines), since, faithful to the Ant build, each Solr module depends on all Solr core, solrj, and example dependencies, and each one of those has to be excluded to thwart transitive dependency resolution. Blech.

          Left to do: verify that generate-maven-artifacts works - I haven't tried it yet.

          Show
          Steve Rowe added a comment - New patch, almost there. Changes: ivy.xml files and ivy-<version>.xml files from the Ivy cache are now parsed with DOM+XPath instead of SAX. All tests pass under the maven build using the filtered POMs. More javadocs added. The filtered grandparent POM is now over 5,000 lines long (previously about 900 lines), since, faithful to the Ant build, each Solr module depends on all Solr core, solrj, and example dependencies, and each one of those has to be excluded to thwart transitive dependency resolution. Blech. Left to do: verify that generate-maven-artifacts works - I haven't tried it yet.
          Hide
          Steve Rowe added a comment - - edited

          Patch, hopefully complete.

          In addition to all tests passing in the Maven build after ant get-maven-poms, generate-maven-artifacts and precommit both pass.

          I'm running ant validate-maven-dependencies and ant nightly-smoke now, and if no problems surface, I'll commit to trunk. I plan on letting it soak for a few days before backporting to branch_4x.

          Show
          Steve Rowe added a comment - - edited Patch, hopefully complete. In addition to all tests passing in the Maven build after ant get-maven-poms , generate-maven-artifacts and precommit both pass. I'm running ant validate-maven-dependencies and ant nightly-smoke now, and if no problems surface, I'll commit to trunk. I plan on letting it soak for a few days before backporting to branch_4x.
          Hide
          Steve Rowe added a comment -

          A last minute change before posting the previous version of the patch, to fix a problem turned up by validate-maven-dependencies - jetty-start was depended on and doesn't have a checksum in solr/licences/ - broke other stuff. That's fixed in this version of the patch.

          Show
          Steve Rowe added a comment - A last minute change before posting the previous version of the patch, to fix a problem turned up by validate-maven-dependencies - jetty-start was depended on and doesn't have a checksum in solr/licences/ - broke other stuff. That's fixed in this version of the patch.
          Hide
          ASF subversion and git services added a comment -

          Commit 1537528 from Steve Rowe in branch 'dev/trunk'
          [ https://svn.apache.org/r1537528 ]

          LUCENE-5217: Maven config: get dependencies from Ant+Ivy; disable transitive dependency resolution for all depended-on artifacts by putting an exclusion for each transitive dependency in the <dependencyManagement> section of the grandparent POM

          Show
          ASF subversion and git services added a comment - Commit 1537528 from Steve Rowe in branch 'dev/trunk' [ https://svn.apache.org/r1537528 ] LUCENE-5217 : Maven config: get dependencies from Ant+Ivy; disable transitive dependency resolution for all depended-on artifacts by putting an exclusion for each transitive dependency in the <dependencyManagement> section of the grandparent POM
          Hide
          ASF subversion and git services added a comment -

          Commit 1537530 from Steve Rowe in branch 'dev/trunk'
          [ https://svn.apache.org/r1537530 ]

          LUCENE-5217: changes entry

          Show
          ASF subversion and git services added a comment - Commit 1537530 from Steve Rowe in branch 'dev/trunk' [ https://svn.apache.org/r1537530 ] LUCENE-5217 : changes entry
          Hide
          Steve Rowe added a comment -

          validate-maven-dependencies and nightly-smoke both passed. Committed to trunk. I'll wait a few days before committing to branch_4x.

          Show
          Steve Rowe added a comment - validate-maven-dependencies and nightly-smoke both passed. Committed to trunk. I'll wait a few days before committing to branch_4x.
          Hide
          ASF subversion and git services added a comment -

          Commit 1541355 from Steve Rowe in branch 'dev/trunk'
          [ https://svn.apache.org/r1541355 ]

          Move LUCENE-5217 and LUCENE-5322 entries to the 4.7 section

          Show
          ASF subversion and git services added a comment - Commit 1541355 from Steve Rowe in branch 'dev/trunk' [ https://svn.apache.org/r1541355 ] Move LUCENE-5217 and LUCENE-5322 entries to the 4.7 section
          Hide
          ASF subversion and git services added a comment -

          Commit 1541357 from Steve Rowe in branch 'dev/branches/branch_4x'
          [ https://svn.apache.org/r1541357 ]

          Backport LUCENE-5217 and LUCENE-5322 to branch_4x

          Show
          ASF subversion and git services added a comment - Commit 1541357 from Steve Rowe in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1541357 ] Backport LUCENE-5217 and LUCENE-5322 to branch_4x

            People

            • Assignee:
              Steve Rowe
              Reporter:
              Robert Muir
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development