Solr
  1. Solr
  2. SOLR-5173

Solr-core's Maven configuration includes test-only Hadoop dependencies as indirect compile-time dependencies

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 4.4
    • Fix Version/s: 4.5, Trunk
    • Component/s: Build
    • Labels:
      None

      Description

      Chris Collins reported on solr-user that solr-core 4.4 has dependencies on hadoop, and indirectly on jetty 6.

      As a workaround for Maven dependencies, the indirect jetty 6 dependency/ies can be excluded.

      The Maven configuration should exclude any compile-time (and also run-time) dependencies that are not Ant/Ivy compile- and run-time dependencies, including jetty 6.

      1. SOLR-5173.patch
        7 kB
        Steve Rowe
      2. SOLR-5173.patch
        5 kB
        Steve Rowe

        Activity

        Steve Rowe created issue -
        Steve Rowe made changes -
        Field Original Value New Value
        Summary Test-only HDFS dependencies are included in release artifacts Test-only Hadoop dependencies are included in release artifacts
        Hide
        Robert Muir added a comment -

        I think we can just remove the jetty and jetty-util 6.1.26 license files from solr/licenses/, since we don't ship those jars.

        Well as far as source release goes, we dont ship any jars. So do we really not care about licensing just because something is used by test code versus source code? Personally I would prefer if we still tracked this and didn't let it become a free-for-all.

        Show
        Robert Muir added a comment - I think we can just remove the jetty and jetty-util 6.1.26 license files from solr/licenses/, since we don't ship those jars. Well as far as source release goes, we dont ship any jars. So do we really not care about licensing just because something is used by test code versus source code? Personally I would prefer if we still tracked this and didn't let it become a free-for-all.
        Hide
        Mark Miller added a comment -

        I'm also +1 on keeping those license files - the src release includes all test stuff, and I think it's just the safe play.

        Show
        Mark Miller added a comment - I'm also +1 on keeping those license files - the src release includes all test stuff, and I think it's just the safe play.
        Hide
        Steve Rowe added a comment -

        Actually, I was wrong about the binary distribution containing Jetty 6 license files - it contains the .jar.sha1 files for Jetty 6, but not the license files.

        In fact, there is a whole bunch of .sha1 files included in the binary distribution that don't need to be there:

        solr-4.4.0/licenses/activation-1.1.jar.sha1
        solr-4.4.0/licenses/AlchemyAPIAnnotator-2.3.1.jar.sha1
        solr-4.4.0/licenses/ant-1.8.2.jar.sha1
        solr-4.4.0/licenses/apache-mime4j-core-0.7.2.jar.sha1
        solr-4.4.0/licenses/apache-mime4j-dom-0.7.2.jar.sha1
        solr-4.4.0/licenses/attributes-binder-1.0.1.jar.sha1
        solr-4.4.0/licenses/bcmail-jdk15-1.45.jar.sha1
        solr-4.4.0/licenses/bcprov-jdk15-1.45.jar.sha1
        solr-4.4.0/licenses/boilerpipe-1.1.0.jar.sha1
        solr-4.4.0/licenses/carrot2-mini-3.6.2.jar.sha1
        solr-4.4.0/licenses/cglib-nodep-2.2.jar.sha1
        solr-4.4.0/licenses/commons-beanutils-1.7.0.jar.sha1
        solr-4.4.0/licenses/commons-cli-1.2.jar.sha1
        solr-4.4.0/licenses/commons-codec-1.7.jar.sha1
        solr-4.4.0/licenses/commons-collections-3.2.1.jar.sha1
        solr-4.4.0/licenses/commons-compress-1.4.1.jar.sha1
        solr-4.4.0/licenses/commons-configuration-1.6.jar.sha1
        solr-4.4.0/licenses/commons-digester-2.0.jar.sha1
        solr-4.4.0/licenses/commons-fileupload-1.2.1.jar.sha1
        solr-4.4.0/licenses/commons-io-2.1.jar.sha1
        solr-4.4.0/licenses/commons-lang-2.6.jar.sha1
        solr-4.4.0/licenses/concurrentlinkedhashmap-lru-1.2.jar.sha1
        solr-4.4.0/licenses/derby-10.9.1.0.jar.sha1
        solr-4.4.0/licenses/dom4j-1.6.1.jar.sha1
        solr-4.4.0/licenses/easymock-3.0.jar.sha1
        solr-4.4.0/licenses/fontbox-1.8.1.jar.sha1
        solr-4.4.0/licenses/guava-14.0.1.jar.sha1
        solr-4.4.0/licenses/hadoop-annotations-2.0.5-alpha.jar.sha1
        solr-4.4.0/licenses/hadoop-auth-2.0.5-alpha.jar.sha1
        solr-4.4.0/licenses/hadoop-common-2.0.5-alpha.jar.sha1
        solr-4.4.0/licenses/hadoop-common-2.0.5-alpha-tests.jar.sha1
        solr-4.4.0/licenses/hadoop-hdfs-2.0.5-alpha.jar.sha1
        solr-4.4.0/licenses/hadoop-hdfs-2.0.5-alpha-tests.jar.sha1
        solr-4.4.0/licenses/hppc-0.4.1.jar.sha1
        solr-4.4.0/licenses/hsqldb-1.8.0.10.jar.sha1
        solr-4.4.0/licenses/httpclient-4.2.3.jar.sha1
        solr-4.4.0/licenses/httpcore-4.2.2.jar.sha1
        solr-4.4.0/licenses/httpmime-4.2.3.jar.sha1
        solr-4.4.0/licenses/icu4j-49.1.jar.sha1
        solr-4.4.0/licenses/isoparser-1.0-RC-1.jar.sha1
        solr-4.4.0/licenses/jackson-core-asl-1.7.4.jar.sha1
        solr-4.4.0/licenses/jackson-mapper-asl-1.7.4.jar.sha1
        solr-4.4.0/licenses/javax.servlet-api-3.0.1.jar.sha1
        solr-4.4.0/licenses/jcl-over-slf4j-1.6.6.jar.sha1
        solr-4.4.0/licenses/jdom-1.0.jar.sha1
        solr-4.4.0/licenses/jempbox-1.8.1.jar.sha1
        solr-4.4.0/licenses/jersey-core-1.16.jar.sha1
        solr-4.4.0/licenses/jetty-6.1.26.jar.sha1
        solr-4.4.0/licenses/jetty-continuation-8.1.10.v20130312.jar.sha1
        solr-4.4.0/licenses/jetty-deploy-8.1.10.v20130312.jar.sha1
        solr-4.4.0/licenses/jetty-http-8.1.10.v20130312.jar.sha1
        solr-4.4.0/licenses/jetty-io-8.1.10.v20130312.jar.sha1
        solr-4.4.0/licenses/jetty-jmx-8.1.10.v20130312.jar.sha1
        solr-4.4.0/licenses/jetty-security-8.1.10.v20130312.jar.sha1
        solr-4.4.0/licenses/jetty-server-8.1.10.v20130312.jar.sha1
        solr-4.4.0/licenses/jetty-servlet-8.1.10.v20130312.jar.sha1
        solr-4.4.0/licenses/jetty-util-6.1.26.jar.sha1
        solr-4.4.0/licenses/jetty-util-8.1.10.v20130312.jar.sha1
        solr-4.4.0/licenses/jetty-webapp-8.1.10.v20130312.jar.sha1
        solr-4.4.0/licenses/jetty-xml-8.1.10.v20130312.jar.sha1
        solr-4.4.0/licenses/joda-time-2.2.jar.sha1
        solr-4.4.0/licenses/jsonic-1.2.7.jar.sha1
        solr-4.4.0/licenses/jul-to-slf4j-1.6.6.jar.sha1
        solr-4.4.0/licenses/junit-4.10.jar.sha1
        solr-4.4.0/licenses/junit4-ant-2.0.10.jar.sha1
        solr-4.4.0/licenses/juniversalchardet-1.0.3.jar.sha1
        solr-4.4.0/licenses/langdetect-1.1-20120112.jar.sha1
        solr-4.4.0/licenses/log4j-1.2.16.jar.sha1
        solr-4.4.0/licenses/mahout-collections-1.0.jar.sha1
        solr-4.4.0/licenses/mahout-math-0.6.jar.sha1
        solr-4.4.0/licenses/mail-1.4.1.jar.sha1
        solr-4.4.0/licenses/metadata-extractor-2.6.2.jar.sha1
        solr-4.4.0/licenses/morfologik-fsa-1.5.5.jar.sha1
        solr-4.4.0/licenses/morfologik-polish-1.5.5.jar.sha1
        solr-4.4.0/licenses/morfologik-stemming-1.5.5.jar.sha1
        solr-4.4.0/licenses/netcdf-4.2-min.jar.sha1
        solr-4.4.0/licenses/noggit-0.5.jar.sha1
        solr-4.4.0/licenses/objenesis-1.2.jar.sha1
        solr-4.4.0/licenses/OpenCalaisAnnotator-2.3.1.jar.sha1
        solr-4.4.0/licenses/org.restlet.ext.servlet-2.1.1.jar.sha1
        solr-4.4.0/licenses/org.restlet-2.1.1.jar.sha1
        solr-4.4.0/licenses/pdfbox-1.8.1.jar.sha1
        solr-4.4.0/licenses/poi-3.9.jar.sha1
        solr-4.4.0/licenses/poi-ooxml-3.9.jar.sha1
        solr-4.4.0/licenses/poi-ooxml-schemas-3.9.jar.sha1
        solr-4.4.0/licenses/poi-scratchpad-3.9.jar.sha1
        solr-4.4.0/licenses/protobuf-java-2.4.0a.jar.sha1
        solr-4.4.0/licenses/randomizedtesting-runner-2.0.10.jar.sha1
        solr-4.4.0/licenses/rome-0.9.jar.sha1
        solr-4.4.0/licenses/servlet-api-3.0.jar.sha1
        solr-4.4.0/licenses/simple-xml-2.6.4.jar.sha1
        solr-4.4.0/licenses/slf4j-api-1.6.6.jar.sha1
        solr-4.4.0/licenses/slf4j-log4j12-1.6.6.jar.sha1
        solr-4.4.0/licenses/spatial4j-0.3.jar.sha1
        solr-4.4.0/licenses/start.jar.sha1
        solr-4.4.0/licenses/Tagger-2.3.1.jar.sha1
        solr-4.4.0/licenses/tagsoup-1.2.1.jar.sha1
        solr-4.4.0/licenses/tika-core-1.4.jar.sha1
        solr-4.4.0/licenses/tika-parsers-1.4.jar.sha1
        solr-4.4.0/licenses/uimaj-core-2.3.1.jar.sha1
        solr-4.4.0/licenses/velocity-1.7.jar.sha1
        solr-4.4.0/licenses/velocity-tools-2.0.jar.sha1
        solr-4.4.0/licenses/vorbis-java-core-0.1.jar.sha1
        solr-4.4.0/licenses/vorbis-java-tika-0.1.jar.sha1
        solr-4.4.0/licenses/WhitespaceTokenizer-2.3.1.jar.sha1
        solr-4.4.0/licenses/wstx-asl-3.2.7.jar.sha1
        solr-4.4.0/licenses/xercesImpl-2.9.1.jar.sha1
        solr-4.4.0/licenses/xmlbeans-2.3.0.jar.sha1
        solr-4.4.0/licenses/xz-1.0.jar.sha1
        solr-4.4.0/licenses/zookeeper-3.4.5.jar.sha1

        Show
        Steve Rowe added a comment - Actually, I was wrong about the binary distribution containing Jetty 6 license files - it contains the .jar.sha1 files for Jetty 6, but not the license files. In fact, there is a whole bunch of .sha1 files included in the binary distribution that don't need to be there: solr-4.4.0/licenses/activation-1.1.jar.sha1 solr-4.4.0/licenses/AlchemyAPIAnnotator-2.3.1.jar.sha1 solr-4.4.0/licenses/ant-1.8.2.jar.sha1 solr-4.4.0/licenses/apache-mime4j-core-0.7.2.jar.sha1 solr-4.4.0/licenses/apache-mime4j-dom-0.7.2.jar.sha1 solr-4.4.0/licenses/attributes-binder-1.0.1.jar.sha1 solr-4.4.0/licenses/bcmail-jdk15-1.45.jar.sha1 solr-4.4.0/licenses/bcprov-jdk15-1.45.jar.sha1 solr-4.4.0/licenses/boilerpipe-1.1.0.jar.sha1 solr-4.4.0/licenses/carrot2-mini-3.6.2.jar.sha1 solr-4.4.0/licenses/cglib-nodep-2.2.jar.sha1 solr-4.4.0/licenses/commons-beanutils-1.7.0.jar.sha1 solr-4.4.0/licenses/commons-cli-1.2.jar.sha1 solr-4.4.0/licenses/commons-codec-1.7.jar.sha1 solr-4.4.0/licenses/commons-collections-3.2.1.jar.sha1 solr-4.4.0/licenses/commons-compress-1.4.1.jar.sha1 solr-4.4.0/licenses/commons-configuration-1.6.jar.sha1 solr-4.4.0/licenses/commons-digester-2.0.jar.sha1 solr-4.4.0/licenses/commons-fileupload-1.2.1.jar.sha1 solr-4.4.0/licenses/commons-io-2.1.jar.sha1 solr-4.4.0/licenses/commons-lang-2.6.jar.sha1 solr-4.4.0/licenses/concurrentlinkedhashmap-lru-1.2.jar.sha1 solr-4.4.0/licenses/derby-10.9.1.0.jar.sha1 solr-4.4.0/licenses/dom4j-1.6.1.jar.sha1 solr-4.4.0/licenses/easymock-3.0.jar.sha1 solr-4.4.0/licenses/fontbox-1.8.1.jar.sha1 solr-4.4.0/licenses/guava-14.0.1.jar.sha1 solr-4.4.0/licenses/hadoop-annotations-2.0.5-alpha.jar.sha1 solr-4.4.0/licenses/hadoop-auth-2.0.5-alpha.jar.sha1 solr-4.4.0/licenses/hadoop-common-2.0.5-alpha.jar.sha1 solr-4.4.0/licenses/hadoop-common-2.0.5-alpha-tests.jar.sha1 solr-4.4.0/licenses/hadoop-hdfs-2.0.5-alpha.jar.sha1 solr-4.4.0/licenses/hadoop-hdfs-2.0.5-alpha-tests.jar.sha1 solr-4.4.0/licenses/hppc-0.4.1.jar.sha1 solr-4.4.0/licenses/hsqldb-1.8.0.10.jar.sha1 solr-4.4.0/licenses/httpclient-4.2.3.jar.sha1 solr-4.4.0/licenses/httpcore-4.2.2.jar.sha1 solr-4.4.0/licenses/httpmime-4.2.3.jar.sha1 solr-4.4.0/licenses/icu4j-49.1.jar.sha1 solr-4.4.0/licenses/isoparser-1.0-RC-1.jar.sha1 solr-4.4.0/licenses/jackson-core-asl-1.7.4.jar.sha1 solr-4.4.0/licenses/jackson-mapper-asl-1.7.4.jar.sha1 solr-4.4.0/licenses/javax.servlet-api-3.0.1.jar.sha1 solr-4.4.0/licenses/jcl-over-slf4j-1.6.6.jar.sha1 solr-4.4.0/licenses/jdom-1.0.jar.sha1 solr-4.4.0/licenses/jempbox-1.8.1.jar.sha1 solr-4.4.0/licenses/jersey-core-1.16.jar.sha1 solr-4.4.0/licenses/jetty-6.1.26.jar.sha1 solr-4.4.0/licenses/jetty-continuation-8.1.10.v20130312.jar.sha1 solr-4.4.0/licenses/jetty-deploy-8.1.10.v20130312.jar.sha1 solr-4.4.0/licenses/jetty-http-8.1.10.v20130312.jar.sha1 solr-4.4.0/licenses/jetty-io-8.1.10.v20130312.jar.sha1 solr-4.4.0/licenses/jetty-jmx-8.1.10.v20130312.jar.sha1 solr-4.4.0/licenses/jetty-security-8.1.10.v20130312.jar.sha1 solr-4.4.0/licenses/jetty-server-8.1.10.v20130312.jar.sha1 solr-4.4.0/licenses/jetty-servlet-8.1.10.v20130312.jar.sha1 solr-4.4.0/licenses/jetty-util-6.1.26.jar.sha1 solr-4.4.0/licenses/jetty-util-8.1.10.v20130312.jar.sha1 solr-4.4.0/licenses/jetty-webapp-8.1.10.v20130312.jar.sha1 solr-4.4.0/licenses/jetty-xml-8.1.10.v20130312.jar.sha1 solr-4.4.0/licenses/joda-time-2.2.jar.sha1 solr-4.4.0/licenses/jsonic-1.2.7.jar.sha1 solr-4.4.0/licenses/jul-to-slf4j-1.6.6.jar.sha1 solr-4.4.0/licenses/junit-4.10.jar.sha1 solr-4.4.0/licenses/junit4-ant-2.0.10.jar.sha1 solr-4.4.0/licenses/juniversalchardet-1.0.3.jar.sha1 solr-4.4.0/licenses/langdetect-1.1-20120112.jar.sha1 solr-4.4.0/licenses/log4j-1.2.16.jar.sha1 solr-4.4.0/licenses/mahout-collections-1.0.jar.sha1 solr-4.4.0/licenses/mahout-math-0.6.jar.sha1 solr-4.4.0/licenses/mail-1.4.1.jar.sha1 solr-4.4.0/licenses/metadata-extractor-2.6.2.jar.sha1 solr-4.4.0/licenses/morfologik-fsa-1.5.5.jar.sha1 solr-4.4.0/licenses/morfologik-polish-1.5.5.jar.sha1 solr-4.4.0/licenses/morfologik-stemming-1.5.5.jar.sha1 solr-4.4.0/licenses/netcdf-4.2-min.jar.sha1 solr-4.4.0/licenses/noggit-0.5.jar.sha1 solr-4.4.0/licenses/objenesis-1.2.jar.sha1 solr-4.4.0/licenses/OpenCalaisAnnotator-2.3.1.jar.sha1 solr-4.4.0/licenses/org.restlet.ext.servlet-2.1.1.jar.sha1 solr-4.4.0/licenses/org.restlet-2.1.1.jar.sha1 solr-4.4.0/licenses/pdfbox-1.8.1.jar.sha1 solr-4.4.0/licenses/poi-3.9.jar.sha1 solr-4.4.0/licenses/poi-ooxml-3.9.jar.sha1 solr-4.4.0/licenses/poi-ooxml-schemas-3.9.jar.sha1 solr-4.4.0/licenses/poi-scratchpad-3.9.jar.sha1 solr-4.4.0/licenses/protobuf-java-2.4.0a.jar.sha1 solr-4.4.0/licenses/randomizedtesting-runner-2.0.10.jar.sha1 solr-4.4.0/licenses/rome-0.9.jar.sha1 solr-4.4.0/licenses/servlet-api-3.0.jar.sha1 solr-4.4.0/licenses/simple-xml-2.6.4.jar.sha1 solr-4.4.0/licenses/slf4j-api-1.6.6.jar.sha1 solr-4.4.0/licenses/slf4j-log4j12-1.6.6.jar.sha1 solr-4.4.0/licenses/spatial4j-0.3.jar.sha1 solr-4.4.0/licenses/start.jar.sha1 solr-4.4.0/licenses/Tagger-2.3.1.jar.sha1 solr-4.4.0/licenses/tagsoup-1.2.1.jar.sha1 solr-4.4.0/licenses/tika-core-1.4.jar.sha1 solr-4.4.0/licenses/tika-parsers-1.4.jar.sha1 solr-4.4.0/licenses/uimaj-core-2.3.1.jar.sha1 solr-4.4.0/licenses/velocity-1.7.jar.sha1 solr-4.4.0/licenses/velocity-tools-2.0.jar.sha1 solr-4.4.0/licenses/vorbis-java-core-0.1.jar.sha1 solr-4.4.0/licenses/vorbis-java-tika-0.1.jar.sha1 solr-4.4.0/licenses/WhitespaceTokenizer-2.3.1.jar.sha1 solr-4.4.0/licenses/wstx-asl-3.2.7.jar.sha1 solr-4.4.0/licenses/xercesImpl-2.9.1.jar.sha1 solr-4.4.0/licenses/xmlbeans-2.3.0.jar.sha1 solr-4.4.0/licenses/xz-1.0.jar.sha1 solr-4.4.0/licenses/zookeeper-3.4.5.jar.sha1
        Hide
        Steve Rowe added a comment -

        Patch fixing Ivy and Maven configurations to treat hadoop-hdfs, hadoop-auth, and hadoop-annotations as test dependencies.

        This causes the following solr-core compilation warnings, presumably because hadoop-common is in the compilation classpath but hadoop-annotations is not - Mark Miller, is this a problem?:

        common.compile-core:
        [mkdir] Created dir: C:\cygwin\home\s\svn\lucene\dev\trunk2\solr\build\solr-core\classes\java
        [javac] Compiling 615 source files to C:\cygwin\home\s\svn\lucene\dev\trunk2\solr\build\solr-core\classes\java
        [javac] C:\cygwin\home\s\svn\lucene\dev\trunk2\solr\core\lib\hadoop-common-2.0.5-alpha.jar(org/apache/hadoop/fs/FileSystem.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate': class file for org.apache.hadoop.classification.InterfaceAudience not found
        [javac] C:\cygwin\home\s\svn\lucene\dev\trunk2\solr\core\lib\hadoop-common-2.0.5-alpha.jar(org/apache/hadoop/fs/FileSystem.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate'
        [javac] C:\cygwin\home\s\svn\lucene\dev\trunk2\solr\core\lib\hadoop-common-2.0.5-alpha.jar(org/apache/hadoop/fs/FileSystem.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate'
        [javac] C:\cygwin\home\s\svn\lucene\dev\trunk2\solr\core\lib\hadoop-common-2.0.5-alpha.jar(org/apache/hadoop/fs/Path.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate'
        [javac] C:\cygwin\home\s\svn\lucene\dev\trunk2\solr\core\lib\hadoop-common-2.0.5-alpha.jar(org/apache/hadoop/security/UserGroupInformation.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate'
        [javac] C:\cygwin\home\s\svn\lucene\dev\trunk2\solr\core\lib\hadoop-common-2.0.5-alpha.jar(org/apache/hadoop/metrics/MetricsUtil.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate'
        [javac] C:\cygwin\home\s\svn\lucene\dev\trunk2\solr\core\lib\hadoop-common-2.0.5-alpha.jar(org/apache/hadoop/metrics/Updater.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate'
        [javac] C:\cygwin\home\s\svn\lucene\dev\trunk2\solr\core\lib\hadoop-common-2.0.5-alpha.jar(org/apache/hadoop/fs/FSDataInputStream.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate'
        [javac] C:\cygwin\home\s\svn\lucene\dev\trunk2\solr\core\lib\hadoop-common-2.0.5-alpha.jar(org/apache/hadoop/fs/FSDataOutputStream.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate'
        [javac] Note: Some input files use or override a deprecated API.
        [javac] Note: Recompile with -Xlint:deprecation for details.
        [javac] Note: Some input files use unchecked or unsafe operations.
        [javac] Note: Recompile with -Xlint:unchecked for details.
        [javac] 9 warnings

        Show
        Steve Rowe added a comment - Patch fixing Ivy and Maven configurations to treat hadoop-hdfs, hadoop-auth, and hadoop-annotations as test dependencies. This causes the following solr-core compilation warnings, presumably because hadoop-common is in the compilation classpath but hadoop-annotations is not - Mark Miller , is this a problem?: common.compile-core: [mkdir] Created dir: C:\cygwin\home\s\svn\lucene\dev\trunk2\solr\build\solr-core\classes\java [javac] Compiling 615 source files to C:\cygwin\home\s\svn\lucene\dev\trunk2\solr\build\solr-core\classes\java [javac] C:\cygwin\home\s\svn\lucene\dev\trunk2\solr\core\lib\hadoop-common-2.0.5-alpha.jar(org/apache/hadoop/fs/FileSystem.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate': class file for org.apache.hadoop.classification.InterfaceAudience not found [javac] C:\cygwin\home\s\svn\lucene\dev\trunk2\solr\core\lib\hadoop-common-2.0.5-alpha.jar(org/apache/hadoop/fs/FileSystem.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate' [javac] C:\cygwin\home\s\svn\lucene\dev\trunk2\solr\core\lib\hadoop-common-2.0.5-alpha.jar(org/apache/hadoop/fs/FileSystem.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate' [javac] C:\cygwin\home\s\svn\lucene\dev\trunk2\solr\core\lib\hadoop-common-2.0.5-alpha.jar(org/apache/hadoop/fs/Path.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate' [javac] C:\cygwin\home\s\svn\lucene\dev\trunk2\solr\core\lib\hadoop-common-2.0.5-alpha.jar(org/apache/hadoop/security/UserGroupInformation.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate' [javac] C:\cygwin\home\s\svn\lucene\dev\trunk2\solr\core\lib\hadoop-common-2.0.5-alpha.jar(org/apache/hadoop/metrics/MetricsUtil.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate' [javac] C:\cygwin\home\s\svn\lucene\dev\trunk2\solr\core\lib\hadoop-common-2.0.5-alpha.jar(org/apache/hadoop/metrics/Updater.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate' [javac] C:\cygwin\home\s\svn\lucene\dev\trunk2\solr\core\lib\hadoop-common-2.0.5-alpha.jar(org/apache/hadoop/fs/FSDataInputStream.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate' [javac] C:\cygwin\home\s\svn\lucene\dev\trunk2\solr\core\lib\hadoop-common-2.0.5-alpha.jar(org/apache/hadoop/fs/FSDataOutputStream.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate' [javac] Note: Some input files use or override a deprecated API. [javac] Note: Recompile with -Xlint:deprecation for details. [javac] Note: Some input files use unchecked or unsafe operations. [javac] Note: Recompile with -Xlint:unchecked for details. [javac] 9 warnings
        Steve Rowe made changes -
        Attachment SOLR-5173.patch [ 12598765 ]
        Hide
        Steve Rowe added a comment -

        From a #solr-dev IRC conversation today:

        10:15 <markrmiller> i think those 3 jars are the compile time dependencies
        10:16               it's the test jars that are not
        10:16               the hdfs common jar requires auth and annotations for whatever reason
        10:16               would be nice if they did something like we do with solrj for the client
        10:17               but they dont
        
        10:22 <sarowe>      compile succeeds with only hadoop-common, leaving out -auth,
        10:22               -annotations, and -hdfs -- there's just a compilation warning about annotations
        10:22               about solrj, I guess you mean a reduced-footprint set of dependencies?
        10:23               runtime deps I mean
        
        10:25 <markrmiller> perhaps its runtime then
        10:25               been many months since I added them
        10:26               but I seem to remember those 3 came in before any test work
        
        10:26 <sarowe>      ok, I'm confused - are you saying that the Ant setup as it is right now,
        10:26               without my patch, is as it should be?
        10:26               right now, the Ant/Ivy/dist setup for solr doesn't make a distinction
        10:27               between compile-time and run-time dependencies
        
        10:27 <markrmiller> not if it was shipping jetty 6
        10:27               I've got to look though
        10:27               the first draft that i committed
        10:27               it was setup right
        10:27               all the test stuff was in the test framework
        10:27               and core only had the min of what it needed
        
        10:27 <sarowe>      jetty 6 did not ship with 4.4
        
        10:27 <markrmiller> i thought that might be the case - its simply a dependency thing?
        
        10:28 <sarowe>      it's a *maven* dependency thing
        10:28               the direct deps for Ivy and Maven are the same
        
        10:28 <markrmiller> not much can be done about it - unless they make a sep client jar
        10:28               maven sucks in the world and thats what you get
        
        10:28 <sarowe>      but Maven also pulls in Jetty 6 as an indirect dep
        
        10:28 <markrmiller> yeah, thats a bummer
        10:28               but nothing we can do
        10:29               thats how maven and the hadoop guys have things
        
        10:29 <sarowe>      well, we can exclude jetty 6 as an indirect dep
        
        10:29 <markrmiller> there is prob a lot you can exclude
        
        10:29 <sarowe>      I did exclude a bunch of stuff
        10:30               but I didn't revisit the exclusions when I moved the test stuff
        10:30               from test-framework to solr-core unfortunately
        
        10:29 <markrmiller> i found the min jars i needed
        
        10:31 <sarowe>      so to confirm about the -hdfs, -auth, and -annotations jars:
        10:31               these are required at run-time, right?
        
        10:31 <markrmiller> as far as I know yes
        10:31               one sec, let me look at an older build
        
        10:32 <sarowe>      thanks - I'll change the patch to keep them as compile-time deps then,
        10:32               since as I said the Ant/Ivy setup doesn't make a distinction between
        10:32               compile-time and run-time deps
        10:32               assuming you find that the older build did that
        
        10:34 <markrmiller> what I added and need for compile/runtime was:
        10:35               common, hdfs, annotations, auth, commons-configuration, protobuf-java,
        10:35               perhaps concurrentlinkedhashmap unless it was already there
        10:35               <dependency org="org.apache.hadoop" name="hadoop-common" rev="${hadoop.version}" transitive="false"/>
        10:35               <dependency org="org.apache.hadoop" name="hadoop-hdfs" rev="${hadoop.version}" transitive="false"/>
        10:35               <dependency org="org.apache.hadoop" name="hadoop-annotations" rev="${hadoop.version}" transitive="false"/>
        10:35               <dependency org="org.apache.hadoop" name="hadoop-auth" rev="${hadoop.version}" transitive="false"/>
        10:35               <dependency org="commons-configuration" name="commons-configuration" rev="1.6" transitive="false"/>
        10:35               <dependency org="com.google.protobuf" name="protobuf-java" rev="2.4.0a" transitive="false"/>
        10:35               <dependency org="com.googlecode.concurrentlinkedhashmap" name="concurrentlinkedhashmap-lru" rev="1.2" transitive="false"/>
        
        Show
        Steve Rowe added a comment - From a #solr-dev IRC conversation today: 10:15 <markrmiller> i think those 3 jars are the compile time dependencies 10:16 it's the test jars that are not 10:16 the hdfs common jar requires auth and annotations for whatever reason 10:16 would be nice if they did something like we do with solrj for the client 10:17 but they dont 10:22 <sarowe> compile succeeds with only hadoop-common, leaving out -auth, 10:22 -annotations, and -hdfs -- there's just a compilation warning about annotations 10:22 about solrj, I guess you mean a reduced-footprint set of dependencies? 10:23 runtime deps I mean 10:25 <markrmiller> perhaps its runtime then 10:25 been many months since I added them 10:26 but I seem to remember those 3 came in before any test work 10:26 <sarowe> ok, I'm confused - are you saying that the Ant setup as it is right now, 10:26 without my patch, is as it should be? 10:26 right now, the Ant/Ivy/dist setup for solr doesn't make a distinction 10:27 between compile-time and run-time dependencies 10:27 <markrmiller> not if it was shipping jetty 6 10:27 I've got to look though 10:27 the first draft that i committed 10:27 it was setup right 10:27 all the test stuff was in the test framework 10:27 and core only had the min of what it needed 10:27 <sarowe> jetty 6 did not ship with 4.4 10:27 <markrmiller> i thought that might be the case - its simply a dependency thing? 10:28 <sarowe> it's a *maven* dependency thing 10:28 the direct deps for Ivy and Maven are the same 10:28 <markrmiller> not much can be done about it - unless they make a sep client jar 10:28 maven sucks in the world and thats what you get 10:28 <sarowe> but Maven also pulls in Jetty 6 as an indirect dep 10:28 <markrmiller> yeah, thats a bummer 10:28 but nothing we can do 10:29 thats how maven and the hadoop guys have things 10:29 <sarowe> well, we can exclude jetty 6 as an indirect dep 10:29 <markrmiller> there is prob a lot you can exclude 10:29 <sarowe> I did exclude a bunch of stuff 10:30 but I didn't revisit the exclusions when I moved the test stuff 10:30 from test-framework to solr-core unfortunately 10:29 <markrmiller> i found the min jars i needed 10:31 <sarowe> so to confirm about the -hdfs, -auth, and -annotations jars: 10:31 these are required at run-time, right? 10:31 <markrmiller> as far as I know yes 10:31 one sec, let me look at an older build 10:32 <sarowe> thanks - I'll change the patch to keep them as compile-time deps then, 10:32 since as I said the Ant/Ivy setup doesn't make a distinction between 10:32 compile-time and run-time deps 10:32 assuming you find that the older build did that 10:34 <markrmiller> what I added and need for compile/runtime was: 10:35 common, hdfs, annotations, auth, commons-configuration, protobuf-java, 10:35 perhaps concurrentlinkedhashmap unless it was already there 10:35 <dependency org="org.apache.hadoop" name="hadoop-common" rev="${hadoop.version}" transitive="false"/> 10:35 <dependency org="org.apache.hadoop" name="hadoop-hdfs" rev="${hadoop.version}" transitive="false"/> 10:35 <dependency org="org.apache.hadoop" name="hadoop-annotations" rev="${hadoop.version}" transitive="false"/> 10:35 <dependency org="org.apache.hadoop" name="hadoop-auth" rev="${hadoop.version}" transitive="false"/> 10:35 <dependency org="commons-configuration" name="commons-configuration" rev="1.6" transitive="false"/> 10:35 <dependency org="com.google.protobuf" name="protobuf-java" rev="2.4.0a" transitive="false"/> 10:35 <dependency org="com.googlecode.concurrentlinkedhashmap" name="concurrentlinkedhashmap-lru" rev="1.2" transitive="false"/>
        Steve Rowe made changes -
        Summary Test-only Hadoop dependencies are included in release artifacts Solr-core's Maven configuration includes test-only Hadoop dependencies as indirect compile-time dependencies
        Steve Rowe made changes -
        Description Chris Collins [reported on solr-user|http://markmail.org/message/evhpcougs5ppafjk] that solr-core 4.4 has dependencies on hadoop, and indirectly on jetty 6.

        As a workaround for Maven dependencies, the hadoop-hdfs, hadoop-auth, and hadoop-annotations dependencies can be excluded, which will also exclude the indirect jetty 6 dependency/ies. hadoop-common is a compile-time dependency, though, so I'm not sure if it's safe to exclude.

        The problems, as far as I can tell, are:

        1) The ivy configuration puts three test-only dependencies (hadoop-hdfs, hadoo-auth, and hadoop-annotations) in solr/core/lib/, rather than where they belong, in solr/core/test-lib/. (hadoop-common is required for solr-core compilation to succeed.)

        2) The Maven configuration makes the equivalent mistake in marking these test-only hadoop dependencies as compile-scope rather than test-scope dependencies.

        3) The Solr .war, which packages everything under solr/core/lib/, includes these three test-only hadoop dependencies (though it does not include any jetty 6 jars).

        4) The license files for jetty and jetty-util v6.1.26, but not the jar files corresponding to them, are included in the Solr distribution.

        I have working (tests pass) local Ant and Maven configurations that treat the three hadoop test-only dependencies properly; as result, the .war will no longer contain them - this will cover problems #1-3 above.

        I think we can just remove the jetty and jetty-util 6.1.26 license files from solr/licenses/, since we don't ship those jars.
        Chris Collins [reported on solr-user|http://markmail.org/message/evhpcougs5ppafjk] that solr-core 4.4 has dependencies on hadoop, and indirectly on jetty 6.

        As a workaround for Maven dependencies, the indirect jetty 6 dependency/ies can be excluded.

        The Maven configuration should exclude any compile-time (and also run-time) dependencies that are not Ant/Ivy compile- and run-time dependencies, including jetty 6.
        Hide
        Steve Rowe added a comment -

        This patch no longer removes hadoop-auth, hadoop-hdfs, and hdfs-annotations as Ivy and Maven dependencies. Instead, the solr-core Maven dependencies are trimmed down to a minimum, and no longer include indirect jetty 6 dependencies.

        Committing shortly.

        Show
        Steve Rowe added a comment - This patch no longer removes hadoop-auth, hadoop-hdfs, and hdfs-annotations as Ivy and Maven dependencies. Instead, the solr-core Maven dependencies are trimmed down to a minimum, and no longer include indirect jetty 6 dependencies. Committing shortly.
        Steve Rowe made changes -
        Attachment SOLR-5173.patch [ 12599251 ]
        Hide
        ASF subversion and git services added a comment -

        Commit 1516264 from Steve Rowe in branch 'dev/trunk'
        [ https://svn.apache.org/r1516264 ]

        SOLR-5173: Solr-core's Maven configuration includes test-only Hadoop dependencies as indirect compile-time dependencies

        Show
        ASF subversion and git services added a comment - Commit 1516264 from Steve Rowe in branch 'dev/trunk' [ https://svn.apache.org/r1516264 ] SOLR-5173 : Solr-core's Maven configuration includes test-only Hadoop dependencies as indirect compile-time dependencies
        Hide
        ASF subversion and git services added a comment -

        Commit 1516273 from Steve Rowe in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1516273 ]

        SOLR-5173: Solr-core's Maven configuration includes test-only Hadoop dependencies as indirect compile-time dependencies (merged trunk r1516264)

        Show
        ASF subversion and git services added a comment - Commit 1516273 from Steve Rowe in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1516273 ] SOLR-5173 : Solr-core's Maven configuration includes test-only Hadoop dependencies as indirect compile-time dependencies (merged trunk r1516264)
        Hide
        Steve Rowe added a comment -

        Committed to trunk and branch_4x.

        Thanks Chris!

        Show
        Steve Rowe added a comment - Committed to trunk and branch_4x. Thanks Chris!
        Steve Rowe made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Fix Version/s 5.0 [ 12321664 ]
        Resolution Fixed [ 1 ]
        Hide
        Adrien Grand added a comment -

        4.5 release -> bulk close

        Show
        Adrien Grand added a comment - 4.5 release -> bulk close
        Adrien Grand made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Steve Rowe
            Reporter:
            Steve Rowe
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development