Lucene - Core
  1. Lucene - Core
  2. LUCENE-5442

Build system should sanity check transative 3rd party dependencies

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.9, 6.0
    • Component/s: general/build
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      SOLR-5365 is an example of a bug that croped up because we upgraded a 3rd party dep (tika) w/o realizing that the version we upgraded too depended on a newer version of another 3rd party dep (commons-compress)

      in a comment in SOLR-5365, Jan suggested that it would be nice if there was an easy way to spot problems like this ... i asked steve about it, thinking maybe this is something the maven build could help with, and he mentioned that there is already an ant task to inspect the ivy transative deps in order to generate the maven deps and it could be used to help detect this sort of problem.

      opening this issue per steve's request as a reminder to look into this possibility.

        Issue Links

          Activity

          Hide
          Mark Miller added a comment -

          We have similar issues with the morphlines stuff (which also heavily uses tika). It's tough because it's extremely hard to have tests that can exercise all of this or to know that you have exercised it all.

          It would be great if you could run something that figured out all the maven transitive version dependencies and let us know what version we are behind on or missing.

          You still have to harmonize, so it won't necessarily be foolproof, but a great check.

          Show
          Mark Miller added a comment - We have similar issues with the morphlines stuff (which also heavily uses tika). It's tough because it's extremely hard to have tests that can exercise all of this or to know that you have exercised it all. It would be great if you could run something that figured out all the maven transitive version dependencies and let us know what version we are behind on or missing. You still have to harmonize, so it won't necessarily be foolproof, but a great check.
          Hide
          Jan Høydahl added a comment -

          It would be great if you could run something that figured out all the maven transitive version dependencies and let us know what version we are behind on or missing.

          Yes, and with Tika specifically we have decided to not include all the possible Parser dependencies jars with Solr, so getting such a report will let us do a qualified decision whether to fix the ivy deps or whether to e.g. change the TikaConfig to exclude certain Parsers from the default setup.

          Show
          Jan Høydahl added a comment - It would be great if you could run something that figured out all the maven transitive version dependencies and let us know what version we are behind on or missing. Yes, and with Tika specifically we have decided to not include all the possible Parser dependencies jars with Solr, so getting such a report will let us do a qualified decision whether to fix the ivy deps or whether to e.g. change the TikaConfig to exclude certain Parsers from the default setup.
          Hide
          Steve Rowe added a comment -

          I've attached a trunk patch that augments the check-lib-versions target to fail the build if a transitive dependency's version is more recent than the corresponding direct dependency's version specified in lucene/ivy-versions.properties. Exceptions are specifiable in a new file lucene/ivy-ignore-conflicts.properties. I've populated this file with the current set of conflicts.

          When I comment out the entries in lucene/ivy-ignore-conflicts.properties, this is the output (and the build fails):

          [libversions] VERSION CONFLICT: transitive dependency in module(s) benchmark, extraction:
          [libversions] /org.apache.commons/commons-compress=1.7
          [libversions] +-- /org.tukaani/xz=1.4 <<< Conflict (direct=1.2)
          [libversions] 
          [libversions] VERSION CONFLICT: transitive dependency in module(s) morphlines-core, map-reduce:
          [libversions] /org.apache.hadoop/hadoop-mapreduce-client-core=2.2.0
          [libversions] +-- /org.apache.hadoop/hadoop-yarn-common=2.2.0
          [libversions]     +-- /org.apache.hadoop/hadoop-yarn-api=2.2.0
          [libversions]         +-- /com.sun.jersey/jersey-json=1.9 <<< Conflict (direct=1.8)
          [libversions] ... and 15 more
          [libversions] 
          [libversions] VERSION CONFLICT: transitive dependency in module(s) morphlines-core, map-reduce:
          [libversions] /org.apache.hadoop/hadoop-mapreduce-client-core=2.2.0
          [libversions] +-- /org.apache.hadoop/hadoop-yarn-common=2.2.0
          [libversions]     +-- /com.sun.jersey/jersey-server=1.9
          [libversions]         +-- /asm/asm=3.2 <<< Conflict (direct=3.1)
          [libversions] ... and 23 more
          [libversions] 
          [libversions] VERSION CONFLICT: transitive dependency in module(s) morphlines-core:
          [libversions] /org.apache.hadoop/hadoop-yarn-server-tests=2.2.0
          [libversions] +-- /io.netty/netty=3.6.2.Final
          [libversions]     +-- /javax.activation/activation=1.1.1 <<< Conflict (direct=1.1)
          [libversions] ... and 14 more
          [libversions] 
          [libversions] VERSION CONFLICT: transitive dependency in module(s) morphlines-core, map-reduce:
          [libversions] /org.apache.hadoop/hadoop-mapreduce-client-core=2.2.0
          [libversions] +-- /org.apache.hadoop/hadoop-yarn-common=2.2.0
          [libversions]     +-- /org.apache.hadoop/hadoop-yarn-api=2.2.0
          [libversions]         +-- /com.sun.jersey.contribs/jersey-guice=1.9 <<< Conflict (direct=1.8)
          [libversions] ... and 13 more
          [libversions] 
          [libversions] VERSION CONFLICT: transitive dependency in module(s) solrj, replicator:
          [libversions] /org.apache.httpcomponents/httpclient=4.3.1
          [libversions] +-- /commons-logging/commons-logging=1.1.3 <<< Conflict (direct=1.1.1)
          [libversions] ... and 1 more
          [libversions] 
          [libversions] VERSION CONFLICT: transitive dependency in module(s) uima:
          [libversions] /org.apache.uima/AlchemyAPIAnnotator=2.3.1
          [libversions] +-- /commons-digester/commons-digester=2.1 <<< Conflict (direct=2.0)
          [libversions] 
          [libversions] VERSION CONFLICT: transitive dependency in module(s) morphlines-core, map-reduce:
          [libversions] /org.apache.hadoop/hadoop-mapreduce-client-core=2.2.0
          [libversions] +-- /org.apache.hadoop/hadoop-common=2.2.0
          [libversions]     +-- /com.sun.jersey/jersey-core=1.9 <<< Conflict (direct=1.8)
          [libversions] ... and 15 more
          [libversions] 
          [libversions] VERSION CONFLICT: transitive dependency in module(s) morphlines-core, map-reduce:
          [libversions] /org.apache.hadoop/hadoop-mapreduce-client-core=2.2.0
          [libversions] +-- /org.apache.hadoop/hadoop-yarn-common=2.2.0
          [libversions]     +-- /org.apache.hadoop/hadoop-yarn-api=2.2.0
          [libversions]         +-- /com.sun.jersey/jersey-json=1.9
          [libversions]             +-- /com.sun.xml.bind/jaxb-impl=2.2.3-1 <<< Conflict (direct=2.2.2)
          [libversions] ... and 23 more
          [libversions] 
          [libversions] VERSION CONFLICT: transitive dependency in module(s) solr-test-framework, core-test-framework:
          [libversions] /com.carrotsearch.randomizedtesting/junit4-ant=2.1.3
          [libversions] +-- /org.ow2.asm/asm=5.0_BETA <<< Conflict (direct=4.1)
          [libversions] 
          [libversions] VERSION CONFLICT: transitive dependency in module(s) morphlines-core, map-reduce:
          [libversions] /org.apache.hadoop/hadoop-mapreduce-client-core=2.2.0
          [libversions] +-- /org.apache.hadoop/hadoop-yarn-common=2.2.0
          [libversions]     +-- /log4j/log4j=1.2.17 <<< Conflict (direct=1.2.16)
          [libversions] ... and 18 more
          [libversions] 
          [libversions] VERSION CONFLICT: transitive dependency in module(s) langid:
          [libversions] /net.arnx/jsonic=1.2.7
          [libversions] +-- /javax.servlet/servlet-api=3.0-alpha-1 <<< Conflict (direct=2.4)
          [libversions] ... and 30 more
          [libversions] 
          [libversions] VERSION CONFLICT: transitive dependency in module(s) uima:
          [libversions] /commons-digester/commons-digester=2.0
          [libversions] +-- /commons-beanutils/commons-beanutils=1.8.0 <<< Conflict (direct=1.7.0)
          [libversions] ... and 1 more
          [libversions] 
          [libversions] VERSION CONFLICT: transitive dependency in module(s) morphlines-core, map-reduce:
          [libversions] /org.apache.hadoop/hadoop-mapreduce-client-core=2.2.0
          [libversions] +-- /org.apache.hadoop/hadoop-yarn-common=2.2.0
          [libversions]     +-- /com.sun.jersey/jersey-server=1.9 <<< Conflict (direct=1.8)
          [libversions] ... and 15 more
          [libversions] 
          [libversions] VERSION CONFLICT: transitive dependency in module(s) solrj:
          [libversions] /org.apache.zookeeper/zookeeper=3.4.6
          [libversions] +-- /io.netty/netty=3.7.0.Final <<< Conflict (direct=3.6.2.Final)
          [libversions] 
          [libversions] VERSION CONFLICT: transitive dependency in module(s) solr-test-framework, core-test-framework:
          [libversions] /com.carrotsearch.randomizedtesting/junit4-ant=2.1.3
          [libversions] +-- /commons-io/commons-io=2.3 <<< Conflict (direct=2.1)
          [libversions] 
          [libversions] VERSION CONFLICT: transitive dependency in module(s) solr-test-framework, core-test-framework:
          [libversions] /com.carrotsearch.randomizedtesting/junit4-ant=2.1.3
          [libversions] +-- /com.google.guava/guava=16.0.1 <<< Conflict (direct=14.0.1)
          [libversions] 
          [libversions] VERSION CONFLICT: transitive dependency in module(s) morphlines-core:
          [libversions] /org.kitesdk/kite-morphlines-avro=0.12.1
          [libversions] +-- /org.apache.avro/avro=1.7.5 <<< Conflict (direct=1.7.4)
          [libversions] 
          [libversions] VERSION CONFLICT: transitive dependency in module(s) morphlines-core:
          [libversions] /org.kitesdk/kite-morphlines-avro=0.12.1
          [libversions] +-- /org.apache.avro/avro=1.7.5
          [libversions]     +-- /org.xerial.snappy/snappy-java=1.0.5 <<< Conflict (direct=1.0.4.1)
          [libversions] ... and 1 more
          [libversions] Checked that ivy-versions.properties and ivy-ignore-conflicts.properties have lexically sorted '/org/name' keys and no duplicates or orphans.
          [libversions] Scanned 44 ivy.xml files for rev="${/org/name}" format.
          [libversions] Found 19 indirect dependency version conflicts.
          [libversions] Completed in 20.55s., 12 error(s).
          
          Show
          Steve Rowe added a comment - I've attached a trunk patch that augments the check-lib-versions target to fail the build if a transitive dependency's version is more recent than the corresponding direct dependency's version specified in lucene/ivy-versions.properties . Exceptions are specifiable in a new file lucene/ivy-ignore-conflicts.properties . I've populated this file with the current set of conflicts. When I comment out the entries in lucene/ivy-ignore-conflicts.properties , this is the output (and the build fails): [libversions] VERSION CONFLICT: transitive dependency in module(s) benchmark, extraction: [libversions] /org.apache.commons/commons-compress=1.7 [libversions] +-- /org.tukaani/xz=1.4 <<< Conflict (direct=1.2) [libversions] [libversions] VERSION CONFLICT: transitive dependency in module(s) morphlines-core, map-reduce: [libversions] /org.apache.hadoop/hadoop-mapreduce-client-core=2.2.0 [libversions] +-- /org.apache.hadoop/hadoop-yarn-common=2.2.0 [libversions] +-- /org.apache.hadoop/hadoop-yarn-api=2.2.0 [libversions] +-- /com.sun.jersey/jersey-json=1.9 <<< Conflict (direct=1.8) [libversions] ... and 15 more [libversions] [libversions] VERSION CONFLICT: transitive dependency in module(s) morphlines-core, map-reduce: [libversions] /org.apache.hadoop/hadoop-mapreduce-client-core=2.2.0 [libversions] +-- /org.apache.hadoop/hadoop-yarn-common=2.2.0 [libversions] +-- /com.sun.jersey/jersey-server=1.9 [libversions] +-- /asm/asm=3.2 <<< Conflict (direct=3.1) [libversions] ... and 23 more [libversions] [libversions] VERSION CONFLICT: transitive dependency in module(s) morphlines-core: [libversions] /org.apache.hadoop/hadoop-yarn-server-tests=2.2.0 [libversions] +-- /io.netty/netty=3.6.2.Final [libversions] +-- /javax.activation/activation=1.1.1 <<< Conflict (direct=1.1) [libversions] ... and 14 more [libversions] [libversions] VERSION CONFLICT: transitive dependency in module(s) morphlines-core, map-reduce: [libversions] /org.apache.hadoop/hadoop-mapreduce-client-core=2.2.0 [libversions] +-- /org.apache.hadoop/hadoop-yarn-common=2.2.0 [libversions] +-- /org.apache.hadoop/hadoop-yarn-api=2.2.0 [libversions] +-- /com.sun.jersey.contribs/jersey-guice=1.9 <<< Conflict (direct=1.8) [libversions] ... and 13 more [libversions] [libversions] VERSION CONFLICT: transitive dependency in module(s) solrj, replicator: [libversions] /org.apache.httpcomponents/httpclient=4.3.1 [libversions] +-- /commons-logging/commons-logging=1.1.3 <<< Conflict (direct=1.1.1) [libversions] ... and 1 more [libversions] [libversions] VERSION CONFLICT: transitive dependency in module(s) uima: [libversions] /org.apache.uima/AlchemyAPIAnnotator=2.3.1 [libversions] +-- /commons-digester/commons-digester=2.1 <<< Conflict (direct=2.0) [libversions] [libversions] VERSION CONFLICT: transitive dependency in module(s) morphlines-core, map-reduce: [libversions] /org.apache.hadoop/hadoop-mapreduce-client-core=2.2.0 [libversions] +-- /org.apache.hadoop/hadoop-common=2.2.0 [libversions] +-- /com.sun.jersey/jersey-core=1.9 <<< Conflict (direct=1.8) [libversions] ... and 15 more [libversions] [libversions] VERSION CONFLICT: transitive dependency in module(s) morphlines-core, map-reduce: [libversions] /org.apache.hadoop/hadoop-mapreduce-client-core=2.2.0 [libversions] +-- /org.apache.hadoop/hadoop-yarn-common=2.2.0 [libversions] +-- /org.apache.hadoop/hadoop-yarn-api=2.2.0 [libversions] +-- /com.sun.jersey/jersey-json=1.9 [libversions] +-- /com.sun.xml.bind/jaxb-impl=2.2.3-1 <<< Conflict (direct=2.2.2) [libversions] ... and 23 more [libversions] [libversions] VERSION CONFLICT: transitive dependency in module(s) solr-test-framework, core-test-framework: [libversions] /com.carrotsearch.randomizedtesting/junit4-ant=2.1.3 [libversions] +-- /org.ow2.asm/asm=5.0_BETA <<< Conflict (direct=4.1) [libversions] [libversions] VERSION CONFLICT: transitive dependency in module(s) morphlines-core, map-reduce: [libversions] /org.apache.hadoop/hadoop-mapreduce-client-core=2.2.0 [libversions] +-- /org.apache.hadoop/hadoop-yarn-common=2.2.0 [libversions] +-- /log4j/log4j=1.2.17 <<< Conflict (direct=1.2.16) [libversions] ... and 18 more [libversions] [libversions] VERSION CONFLICT: transitive dependency in module(s) langid: [libversions] /net.arnx/jsonic=1.2.7 [libversions] +-- /javax.servlet/servlet-api=3.0-alpha-1 <<< Conflict (direct=2.4) [libversions] ... and 30 more [libversions] [libversions] VERSION CONFLICT: transitive dependency in module(s) uima: [libversions] /commons-digester/commons-digester=2.0 [libversions] +-- /commons-beanutils/commons-beanutils=1.8.0 <<< Conflict (direct=1.7.0) [libversions] ... and 1 more [libversions] [libversions] VERSION CONFLICT: transitive dependency in module(s) morphlines-core, map-reduce: [libversions] /org.apache.hadoop/hadoop-mapreduce-client-core=2.2.0 [libversions] +-- /org.apache.hadoop/hadoop-yarn-common=2.2.0 [libversions] +-- /com.sun.jersey/jersey-server=1.9 <<< Conflict (direct=1.8) [libversions] ... and 15 more [libversions] [libversions] VERSION CONFLICT: transitive dependency in module(s) solrj: [libversions] /org.apache.zookeeper/zookeeper=3.4.6 [libversions] +-- /io.netty/netty=3.7.0.Final <<< Conflict (direct=3.6.2.Final) [libversions] [libversions] VERSION CONFLICT: transitive dependency in module(s) solr-test-framework, core-test-framework: [libversions] /com.carrotsearch.randomizedtesting/junit4-ant=2.1.3 [libversions] +-- /commons-io/commons-io=2.3 <<< Conflict (direct=2.1) [libversions] [libversions] VERSION CONFLICT: transitive dependency in module(s) solr-test-framework, core-test-framework: [libversions] /com.carrotsearch.randomizedtesting/junit4-ant=2.1.3 [libversions] +-- /com.google.guava/guava=16.0.1 <<< Conflict (direct=14.0.1) [libversions] [libversions] VERSION CONFLICT: transitive dependency in module(s) morphlines-core: [libversions] /org.kitesdk/kite-morphlines-avro=0.12.1 [libversions] +-- /org.apache.avro/avro=1.7.5 <<< Conflict (direct=1.7.4) [libversions] [libversions] VERSION CONFLICT: transitive dependency in module(s) morphlines-core: [libversions] /org.kitesdk/kite-morphlines-avro=0.12.1 [libversions] +-- /org.apache.avro/avro=1.7.5 [libversions] +-- /org.xerial.snappy/snappy-java=1.0.5 <<< Conflict (direct=1.0.4.1) [libversions] ... and 1 more [libversions] Checked that ivy-versions.properties and ivy-ignore-conflicts.properties have lexically sorted '/org/name' keys and no duplicates or orphans. [libversions] Scanned 44 ivy.xml files for rev="${/org/name}" format. [libversions] Found 19 indirect dependency version conflicts. [libversions] Completed in 20.55s., 12 error(s).
          Hide
          Steve Rowe added a comment -

          I think this is ready to go. If there are no objections, I'll commit tomorrow.

          Show
          Steve Rowe added a comment - I think this is ready to go. If there are no objections, I'll commit tomorrow.
          Hide
          Mark Miller added a comment -

          Thank you Steve! This is so useful.

          Show
          Mark Miller added a comment - Thank you Steve! This is so useful.
          Hide
          ASF subversion and git services added a comment -

          Commit 1598538 from Steve Rowe in branch 'dev/trunk'
          [ https://svn.apache.org/r1598538 ]

          LUCENE-5442: ant check-lib-versions will fail the build if there are unexpected version conflicts between direct and transitive dependencies.

          Show
          ASF subversion and git services added a comment - Commit 1598538 from Steve Rowe in branch 'dev/trunk' [ https://svn.apache.org/r1598538 ] LUCENE-5442 : ant check-lib-versions will fail the build if there are unexpected version conflicts between direct and transitive dependencies.
          Hide
          ASF subversion and git services added a comment -

          Commit 1598539 from Steve Rowe in branch 'dev/branches/branch_4x'
          [ https://svn.apache.org/r1598539 ]

          LUCENE-5442: ant check-lib-versions will fail the build if there are unexpected version conflicts between direct and transitive dependencies. (merged trunk r1598538)

          Show
          ASF subversion and git services added a comment - Commit 1598539 from Steve Rowe in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1598539 ] LUCENE-5442 : ant check-lib-versions will fail the build if there are unexpected version conflicts between direct and transitive dependencies. (merged trunk r1598538)
          Hide
          Steve Rowe added a comment -

          Committed to trunk and branch_4x.

          I'll open a follow-on issue to reduce the number of expected version conflicts listed in ivy-ignore-conflicts.properties, by upgrading the corresponding direct dependencies in ivy-versions.properties.

          Show
          Steve Rowe added a comment - Committed to trunk and branch_4x. I'll open a follow-on issue to reduce the number of expected version conflicts listed in ivy-ignore-conflicts.properties , by upgrading the corresponding direct dependencies in ivy-versions.properties .
          Hide
          Steve Rowe added a comment -

          I'll open a follow-on issue to reduce the number of expected version conflicts listed in ivy-ignore-conflicts.properties, by upgrading the corresponding direct dependencies in ivy-versions.properties.

          Done: LUCENE-5715

          Show
          Steve Rowe added a comment - I'll open a follow-on issue to reduce the number of expected version conflicts listed in ivy-ignore-conflicts.properties , by upgrading the corresponding direct dependencies in ivy-versions.properties . Done: LUCENE-5715
          Hide
          ASF subversion and git services added a comment -

          Commit 1599134 from Steve Rowe in branch 'dev/trunk'
          [ https://svn.apache.org/r1599134 ]

          LUCENE-5442: smoke tester: 'ivy-ignore-conflicts.properties' is not a foreign invader

          Show
          ASF subversion and git services added a comment - Commit 1599134 from Steve Rowe in branch 'dev/trunk' [ https://svn.apache.org/r1599134 ] LUCENE-5442 : smoke tester: 'ivy-ignore-conflicts.properties' is not a foreign invader
          Hide
          ASF subversion and git services added a comment -

          Commit 1599138 from Steve Rowe in branch 'dev/branches/branch_4x'
          [ https://svn.apache.org/r1599138 ]

          LUCENE-5442: smoke tester: 'ivy-ignore-conflicts.properties' is not a foreign invader (merge trunk r1599134)

          Show
          ASF subversion and git services added a comment - Commit 1599138 from Steve Rowe in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1599138 ] LUCENE-5442 : smoke tester: 'ivy-ignore-conflicts.properties' is not a foreign invader (merge trunk r1599134)

            People

            • Assignee:
              Steve Rowe
              Reporter:
              Hoss Man
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development