Lucene - Core
  1. Lucene - Core
  2. LUCENE-5273

Binary artifacts in Lucene and Solr convenience binary distributions accompanying a release, including on Maven Central, should be identical across all distributions

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.6, 6.0
    • Component/s: general/build
    • Labels:
      None
    • Lucene Fields:
      New, Patch Available

      Description

      As mentioned in various issues (e.g. LUCENE-3655, LUCENE-3885, SOLR-4766), we release multiple versions of the same artifact: binary Maven artifacts are not identical to the ones in the Lucene and Solr binary distributions, and the Lucene jars in the Solr binary distribution, including within the war, are not identical to the ones in the Lucene binary distribution. This is bad.

      It's (probably always?) not horribly bad, since the differences all appear to be caused by the build re-creating manifests and re-building jars and the Solr war from their constituents at various points in the release build process; as a result, manifest timestamp attributes, as well as archive metadata (at least constituent timestamps, maybe other things?), differ each time a jar is rebuilt.

      1. LUCENE-5273.patch
        26 kB
        Steve Rowe
      2. LUCENE-5273-optimized.patch
        2 kB
        Uwe Schindler
      3. LUCENE-5273-slowdown-fixed.patch
        7 kB
        Steve Rowe

        Activity

        Hide
        Steve Rowe added a comment -

        Patch that fixes the problem, and adds checks to smokeTestRelease.py.

        I had been thinking that the way to fix this problem was to somehow not rebuild manifests and jars when their constituents had not changed. A couple efforts in this direction failed for various reasons, though.

        Instead, this patch causes the release building process to unpack lucene-<version>.tgz to lucene/build/lucene.tgz.unpacked, and solr-<version>.tgz to solr/build/solr.tgz.unpacked, and then pull binary artifacts from the unpacked lucene distribution when they're needed in the Solr distribution and the Solr war, and from the appropriate unpacked distribution when deploying Lucene and Solr Maven artifacts.

        ant nightly-smoke succeeds on trunk with this patch.

        I backported the smokeTestRelease.py changes to the lucene_solr_4_5 branch, then ran the smoke tester against the final 4.5 RC, after first converting exceptions for non-identicality to print statements with "Should fail:" prepended, and I got the following output:

          unpack solr-4.5.0.tgz...
            verify JAR metadata/identity/no javax.* or java.* classes...
            unpack lucene-4.5.0.tgz...
        Should fail: Artifact /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/contrib/analysis-extras/lucene-libs/lucene-analyzers-icu-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/icu/lucene-analyzers-icu-4.5.0.jar in Lucene binary distribution
        Should fail: Artifact /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/contrib/analysis-extras/lucene-libs/lucene-analyzers-morfologik-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/morfologik/lucene-analyzers-morfologik-4.5.0.jar in Lucene binary distribution
        Should fail: Artifact /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/contrib/analysis-extras/lucene-libs/lucene-analyzers-smartcn-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/smartcn/lucene-analyzers-smartcn-4.5.0.jar in Lucene binary distribution
        Should fail: Artifact /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/contrib/analysis-extras/lucene-libs/lucene-analyzers-stempel-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/stempel/lucene-analyzers-stempel-4.5.0.jar in Lucene binary distribution
              **WARNING**: skipping check of /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/contrib/dataimporthandler/lib/activation-1.1.jar: it has javax.* classes
              **WARNING**: skipping check of /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/contrib/dataimporthandler/lib/mail-1.4.1.jar: it has javax.* classes
        Should fail: Artifact /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/contrib/uima/lucene-libs/lucene-analyzers-uima-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/uima/lucene-analyzers-uima-4.5.0.jar in Lucene binary distribution
        Should fail: Artifact /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/dist/test-framework/lucene-libs/lucene-test-framework-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/test-framework/lucene-test-framework-4.5.0.jar in Lucene binary distribution
            verify WAR metadata/contained JAR identity/no javax.* or java.* classes...
            unpack lucene-4.5.0.tgz...
        Should fail: Artifact WEB-INF/lib/lucene-analyzers-common-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/common/lucene-analyzers-common-4.5.0.jar in Lucene binary distribution
        Should fail: Artifact WEB-INF/lib/lucene-analyzers-kuromoji-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/kuromoji/lucene-analyzers-kuromoji-4.5.0.jar in Lucene binary distribution
        Should fail: Artifact WEB-INF/lib/lucene-analyzers-phonetic-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/phonetic/lucene-analyzers-phonetic-4.5.0.jar in Lucene binary distribution
        Should fail: Artifact WEB-INF/lib/lucene-codecs-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/codecs/lucene-codecs-4.5.0.jar in Lucene binary distribution
        Should fail: Artifact WEB-INF/lib/lucene-core-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/core/lucene-core-4.5.0.jar in Lucene binary distribution
        Should fail: Artifact WEB-INF/lib/lucene-grouping-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/grouping/lucene-grouping-4.5.0.jar in Lucene binary distribution
        Should fail: Artifact WEB-INF/lib/lucene-highlighter-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/highlighter/lucene-highlighter-4.5.0.jar in Lucene binary distribution
        Should fail: Artifact WEB-INF/lib/lucene-join-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/join/lucene-join-4.5.0.jar in Lucene binary distribution
        Should fail: Artifact WEB-INF/lib/lucene-memory-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/memory/lucene-memory-4.5.0.jar in Lucene binary distribution
        Should fail: Artifact WEB-INF/lib/lucene-misc-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/misc/lucene-misc-4.5.0.jar in Lucene binary distribution
        Should fail: Artifact WEB-INF/lib/lucene-queries-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/queries/lucene-queries-4.5.0.jar in Lucene binary distribution
        Should fail: Artifact WEB-INF/lib/lucene-queryparser-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/queryparser/lucene-queryparser-4.5.0.jar in Lucene binary distribution
        Should fail: Artifact WEB-INF/lib/lucene-spatial-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/spatial/lucene-spatial-4.5.0.jar in Lucene binary distribution
        Should fail: Artifact WEB-INF/lib/lucene-suggest-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/suggest/lucene-suggest-4.5.0.jar in Lucene binary distribution
        [...]
          unpack solr-4.5.0.zip...
            verify JAR metadata/identity/no javax.* or java.* classes...
            unpack lucene-4.5.0.tgz...
        Should fail: Artifact /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/contrib/analysis-extras/lucene-libs/lucene-analyzers-icu-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/icu/lucene-analyzers-icu-4.5.0.jar in Lucene binary distribution
        Should fail: Artifact /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/contrib/analysis-extras/lucene-libs/lucene-analyzers-morfologik-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/morfologik/lucene-analyzers-morfologik-4.5.0.jar in Lucene binary distribution
        Should fail: Artifact /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/contrib/analysis-extras/lucene-libs/lucene-analyzers-smartcn-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/smartcn/lucene-analyzers-smartcn-4.5.0.jar in Lucene binary distribution
        Should fail: Artifact /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/contrib/analysis-extras/lucene-libs/lucene-analyzers-stempel-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/stempel/lucene-analyzers-stempel-4.5.0.jar in Lucene binary distribution
              **WARNING**: skipping check of /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/contrib/dataimporthandler/lib/activation-1.1.jar: it has javax.* classes
              **WARNING**: skipping check of /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/contrib/dataimporthandler/lib/mail-1.4.1.jar: it has javax.* classes
        Should fail: Artifact /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/contrib/uima/lucene-libs/lucene-analyzers-uima-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/uima/lucene-analyzers-uima-4.5.0.jar in Lucene binary distribution
        Should fail: Artifact /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/dist/test-framework/lucene-libs/lucene-test-framework-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/test-framework/lucene-test-framework-4.5.0.jar in Lucene binary distribution
            verify WAR metadata/contained JAR identity/no javax.* or java.* classes...
            unpack lucene-4.5.0.tgz...
        Should fail: Artifact WEB-INF/lib/lucene-analyzers-common-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/common/lucene-analyzers-common-4.5.0.jar in Lucene binary distribution
        Should fail: Artifact WEB-INF/lib/lucene-analyzers-kuromoji-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/kuromoji/lucene-analyzers-kuromoji-4.5.0.jar in Lucene binary distribution
        Should fail: Artifact WEB-INF/lib/lucene-analyzers-phonetic-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/phonetic/lucene-analyzers-phonetic-4.5.0.jar in Lucene binary distribution
        Should fail: Artifact WEB-INF/lib/lucene-codecs-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/codecs/lucene-codecs-4.5.0.jar in Lucene binary distribution
        Should fail: Artifact WEB-INF/lib/lucene-core-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/core/lucene-core-4.5.0.jar in Lucene binary distribution
        Should fail: Artifact WEB-INF/lib/lucene-grouping-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/grouping/lucene-grouping-4.5.0.jar in Lucene binary distribution
        Should fail: Artifact WEB-INF/lib/lucene-highlighter-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/highlighter/lucene-highlighter-4.5.0.jar in Lucene binary distribution
        Should fail: Artifact WEB-INF/lib/lucene-join-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/join/lucene-join-4.5.0.jar in Lucene binary distribution
        Should fail: Artifact WEB-INF/lib/lucene-memory-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/memory/lucene-memory-4.5.0.jar in Lucene binary distribution
        Should fail: Artifact WEB-INF/lib/lucene-misc-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/misc/lucene-misc-4.5.0.jar in Lucene binary distribution
        Should fail: Artifact WEB-INF/lib/lucene-queries-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/queries/lucene-queries-4.5.0.jar in Lucene binary distribution
        Should fail: Artifact WEB-INF/lib/lucene-queryparser-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/queryparser/lucene-queryparser-4.5.0.jar in Lucene binary distribution
        Should fail: Artifact WEB-INF/lib/lucene-spatial-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/spatial/lucene-spatial-4.5.0.jar in Lucene binary distribution
        Should fail: Artifact WEB-INF/lib/lucene-suggest-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/suggest/lucene-suggest-4.5.0.jar in Lucene binary distribution
        [...]
        Test Maven artifacts for Lucene and Solr...
        [...]
            verify that Maven artifacts are same as in the binary distribution...
        Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-analyzers-common/4.5.0/lucene-analyzers-common-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/common/lucene-analyzers-common-4.5.0.jar in lucene binary distribution
        Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-analyzers-icu/4.5.0/lucene-analyzers-icu-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/icu/lucene-analyzers-icu-4.5.0.jar in lucene binary distribution
        Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-analyzers-kuromoji/4.5.0/lucene-analyzers-kuromoji-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/kuromoji/lucene-analyzers-kuromoji-4.5.0.jar in lucene binary distribution
        Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-analyzers-morfologik/4.5.0/lucene-analyzers-morfologik-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/morfologik/lucene-analyzers-morfologik-4.5.0.jar in lucene binary distribution
        Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-analyzers-phonetic/4.5.0/lucene-analyzers-phonetic-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/phonetic/lucene-analyzers-phonetic-4.5.0.jar in lucene binary distribution
        Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-analyzers-smartcn/4.5.0/lucene-analyzers-smartcn-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/smartcn/lucene-analyzers-smartcn-4.5.0.jar in lucene binary distribution
        Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-analyzers-stempel/4.5.0/lucene-analyzers-stempel-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/stempel/lucene-analyzers-stempel-4.5.0.jar in lucene binary distribution
        Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-analyzers-uima/4.5.0/lucene-analyzers-uima-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/uima/lucene-analyzers-uima-4.5.0.jar in lucene binary distribution
        Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-benchmark/4.5.0/lucene-benchmark-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/benchmark/lucene-benchmark-4.5.0.jar in lucene binary distribution
        Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-classification/4.5.0/lucene-classification-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/classification/lucene-classification-4.5.0.jar in lucene binary distribution
        Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-codecs/4.5.0/lucene-codecs-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/codecs/lucene-codecs-4.5.0.jar in lucene binary distribution
        Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-core/4.5.0/lucene-core-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/core/lucene-core-4.5.0.jar in lucene binary distribution
        Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-demo/4.5.0/lucene-demo-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/demo/lucene-demo-4.5.0.jar in lucene binary distribution
        Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-facet/4.5.0/lucene-facet-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/facet/lucene-facet-4.5.0.jar in lucene binary distribution
        Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-grouping/4.5.0/lucene-grouping-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/grouping/lucene-grouping-4.5.0.jar in lucene binary distribution
        Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-highlighter/4.5.0/lucene-highlighter-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/highlighter/lucene-highlighter-4.5.0.jar in lucene binary distribution
        Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-join/4.5.0/lucene-join-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/join/lucene-join-4.5.0.jar in lucene binary distribution
        Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-memory/4.5.0/lucene-memory-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/memory/lucene-memory-4.5.0.jar in lucene binary distribution
        Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-misc/4.5.0/lucene-misc-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/misc/lucene-misc-4.5.0.jar in lucene binary distribution
        Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-queries/4.5.0/lucene-queries-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/queries/lucene-queries-4.5.0.jar in lucene binary distribution
        Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-queryparser/4.5.0/lucene-queryparser-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/queryparser/lucene-queryparser-4.5.0.jar in lucene binary distribution
        Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-replicator/4.5.0/lucene-replicator-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/replicator/lucene-replicator-4.5.0.jar in lucene binary distribution
        Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-sandbox/4.5.0/lucene-sandbox-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/sandbox/lucene-sandbox-4.5.0.jar in lucene binary distribution
        Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-spatial/4.5.0/lucene-spatial-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/spatial/lucene-spatial-4.5.0.jar in lucene binary distribution
        Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-suggest/4.5.0/lucene-suggest-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/suggest/lucene-suggest-4.5.0.jar in lucene binary distribution
        Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-test-framework/4.5.0/lucene-test-framework-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/test-framework/lucene-test-framework-4.5.0.jar in lucene binary distribution
        Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/solr/solr-analysis-extras/4.5.0/solr-analysis-extras-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-solr-getBinaryDistFiles/solr-4.5.0/dist/solr-analysis-extras-4.5.0.jar in solr binary distribution
        Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/solr/solr-cell/4.5.0/solr-cell-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-solr-getBinaryDistFiles/solr-4.5.0/dist/solr-cell-4.5.0.jar in solr binary distribution
        Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/solr/solr-clustering/4.5.0/solr-clustering-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-solr-getBinaryDistFiles/solr-4.5.0/dist/solr-clustering-4.5.0.jar in solr binary distribution
        Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/solr/solr-core/4.5.0/solr-core-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-solr-getBinaryDistFiles/solr-4.5.0/dist/solr-core-4.5.0.jar in solr binary distribution
        Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/solr/solr-dataimporthandler-extras/4.5.0/solr-dataimporthandler-extras-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-solr-getBinaryDistFiles/solr-4.5.0/dist/solr-dataimporthandler-extras-4.5.0.jar in solr binary distribution
        Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/solr/solr-dataimporthandler/4.5.0/solr-dataimporthandler-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-solr-getBinaryDistFiles/solr-4.5.0/dist/solr-dataimporthandler-4.5.0.jar in solr binary distribution
        Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/solr/solr-langid/4.5.0/solr-langid-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-solr-getBinaryDistFiles/solr-4.5.0/dist/solr-langid-4.5.0.jar in solr binary distribution
        Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/solr/solr-solrj/4.5.0/solr-solrj-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-solr-getBinaryDistFiles/solr-4.5.0/dist/solr-solrj-4.5.0.jar in solr binary distribution
        Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/solr/solr-test-framework/4.5.0/solr-test-framework-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-solr-getBinaryDistFiles/solr-4.5.0/dist/solr-test-framework-4.5.0.jar in solr binary distribution
        Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/solr/solr-uima/4.5.0/solr-uima-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-solr-getBinaryDistFiles/solr-4.5.0/dist/solr-uima-4.5.0.jar in solr binary distribution
        Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/solr/solr-velocity/4.5.0/solr-velocity-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-solr-getBinaryDistFiles/solr-4.5.0/dist/solr-velocity-4.5.0.jar in solr binary distribution
        Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/solr/solr/4.5.0/solr-4.5.0.war is not identical to /Users/sarowe/temp/tmpDir/unpack-solr-getBinaryDistFiles/solr-4.5.0/dist/solr-4.5.0.war in solr binary distribution
        

        So I think it works.

        Show
        Steve Rowe added a comment - Patch that fixes the problem, and adds checks to smokeTestRelease.py . I had been thinking that the way to fix this problem was to somehow not rebuild manifests and jars when their constituents had not changed. A couple efforts in this direction failed for various reasons, though. Instead, this patch causes the release building process to unpack lucene-<version>.tgz to lucene/build/lucene.tgz.unpacked , and solr-<version>.tgz to solr/build/solr.tgz.unpacked , and then pull binary artifacts from the unpacked lucene distribution when they're needed in the Solr distribution and the Solr war, and from the appropriate unpacked distribution when deploying Lucene and Solr Maven artifacts. ant nightly-smoke succeeds on trunk with this patch. I backported the smokeTestRelease.py changes to the lucene_solr_4_5 branch, then ran the smoke tester against the final 4.5 RC, after first converting exceptions for non-identicality to print statements with "Should fail:" prepended, and I got the following output: unpack solr-4.5.0.tgz... verify JAR metadata/identity/no javax.* or java.* classes... unpack lucene-4.5.0.tgz... Should fail: Artifact /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/contrib/analysis-extras/lucene-libs/lucene-analyzers-icu-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/icu/lucene-analyzers-icu-4.5.0.jar in Lucene binary distribution Should fail: Artifact /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/contrib/analysis-extras/lucene-libs/lucene-analyzers-morfologik-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/morfologik/lucene-analyzers-morfologik-4.5.0.jar in Lucene binary distribution Should fail: Artifact /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/contrib/analysis-extras/lucene-libs/lucene-analyzers-smartcn-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/smartcn/lucene-analyzers-smartcn-4.5.0.jar in Lucene binary distribution Should fail: Artifact /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/contrib/analysis-extras/lucene-libs/lucene-analyzers-stempel-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/stempel/lucene-analyzers-stempel-4.5.0.jar in Lucene binary distribution **WARNING**: skipping check of /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/contrib/dataimporthandler/lib/activation-1.1.jar: it has javax.* classes **WARNING**: skipping check of /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/contrib/dataimporthandler/lib/mail-1.4.1.jar: it has javax.* classes Should fail: Artifact /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/contrib/uima/lucene-libs/lucene-analyzers-uima-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/uima/lucene-analyzers-uima-4.5.0.jar in Lucene binary distribution Should fail: Artifact /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/dist/test-framework/lucene-libs/lucene-test-framework-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/test-framework/lucene-test-framework-4.5.0.jar in Lucene binary distribution verify WAR metadata/contained JAR identity/no javax.* or java.* classes... unpack lucene-4.5.0.tgz... Should fail: Artifact WEB-INF/lib/lucene-analyzers-common-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/common/lucene-analyzers-common-4.5.0.jar in Lucene binary distribution Should fail: Artifact WEB-INF/lib/lucene-analyzers-kuromoji-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/kuromoji/lucene-analyzers-kuromoji-4.5.0.jar in Lucene binary distribution Should fail: Artifact WEB-INF/lib/lucene-analyzers-phonetic-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/phonetic/lucene-analyzers-phonetic-4.5.0.jar in Lucene binary distribution Should fail: Artifact WEB-INF/lib/lucene-codecs-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/codecs/lucene-codecs-4.5.0.jar in Lucene binary distribution Should fail: Artifact WEB-INF/lib/lucene-core-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/core/lucene-core-4.5.0.jar in Lucene binary distribution Should fail: Artifact WEB-INF/lib/lucene-grouping-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/grouping/lucene-grouping-4.5.0.jar in Lucene binary distribution Should fail: Artifact WEB-INF/lib/lucene-highlighter-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/highlighter/lucene-highlighter-4.5.0.jar in Lucene binary distribution Should fail: Artifact WEB-INF/lib/lucene-join-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/join/lucene-join-4.5.0.jar in Lucene binary distribution Should fail: Artifact WEB-INF/lib/lucene-memory-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/memory/lucene-memory-4.5.0.jar in Lucene binary distribution Should fail: Artifact WEB-INF/lib/lucene-misc-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/misc/lucene-misc-4.5.0.jar in Lucene binary distribution Should fail: Artifact WEB-INF/lib/lucene-queries-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/queries/lucene-queries-4.5.0.jar in Lucene binary distribution Should fail: Artifact WEB-INF/lib/lucene-queryparser-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/queryparser/lucene-queryparser-4.5.0.jar in Lucene binary distribution Should fail: Artifact WEB-INF/lib/lucene-spatial-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/spatial/lucene-spatial-4.5.0.jar in Lucene binary distribution Should fail: Artifact WEB-INF/lib/lucene-suggest-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/suggest/lucene-suggest-4.5.0.jar in Lucene binary distribution [...] unpack solr-4.5.0.zip... verify JAR metadata/identity/no javax.* or java.* classes... unpack lucene-4.5.0.tgz... Should fail: Artifact /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/contrib/analysis-extras/lucene-libs/lucene-analyzers-icu-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/icu/lucene-analyzers-icu-4.5.0.jar in Lucene binary distribution Should fail: Artifact /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/contrib/analysis-extras/lucene-libs/lucene-analyzers-morfologik-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/morfologik/lucene-analyzers-morfologik-4.5.0.jar in Lucene binary distribution Should fail: Artifact /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/contrib/analysis-extras/lucene-libs/lucene-analyzers-smartcn-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/smartcn/lucene-analyzers-smartcn-4.5.0.jar in Lucene binary distribution Should fail: Artifact /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/contrib/analysis-extras/lucene-libs/lucene-analyzers-stempel-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/stempel/lucene-analyzers-stempel-4.5.0.jar in Lucene binary distribution **WARNING**: skipping check of /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/contrib/dataimporthandler/lib/activation-1.1.jar: it has javax.* classes **WARNING**: skipping check of /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/contrib/dataimporthandler/lib/mail-1.4.1.jar: it has javax.* classes Should fail: Artifact /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/contrib/uima/lucene-libs/lucene-analyzers-uima-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/uima/lucene-analyzers-uima-4.5.0.jar in Lucene binary distribution Should fail: Artifact /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/dist/test-framework/lucene-libs/lucene-test-framework-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/test-framework/lucene-test-framework-4.5.0.jar in Lucene binary distribution verify WAR metadata/contained JAR identity/no javax.* or java.* classes... unpack lucene-4.5.0.tgz... Should fail: Artifact WEB-INF/lib/lucene-analyzers-common-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/common/lucene-analyzers-common-4.5.0.jar in Lucene binary distribution Should fail: Artifact WEB-INF/lib/lucene-analyzers-kuromoji-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/kuromoji/lucene-analyzers-kuromoji-4.5.0.jar in Lucene binary distribution Should fail: Artifact WEB-INF/lib/lucene-analyzers-phonetic-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/phonetic/lucene-analyzers-phonetic-4.5.0.jar in Lucene binary distribution Should fail: Artifact WEB-INF/lib/lucene-codecs-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/codecs/lucene-codecs-4.5.0.jar in Lucene binary distribution Should fail: Artifact WEB-INF/lib/lucene-core-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/core/lucene-core-4.5.0.jar in Lucene binary distribution Should fail: Artifact WEB-INF/lib/lucene-grouping-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/grouping/lucene-grouping-4.5.0.jar in Lucene binary distribution Should fail: Artifact WEB-INF/lib/lucene-highlighter-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/highlighter/lucene-highlighter-4.5.0.jar in Lucene binary distribution Should fail: Artifact WEB-INF/lib/lucene-join-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/join/lucene-join-4.5.0.jar in Lucene binary distribution Should fail: Artifact WEB-INF/lib/lucene-memory-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/memory/lucene-memory-4.5.0.jar in Lucene binary distribution Should fail: Artifact WEB-INF/lib/lucene-misc-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/misc/lucene-misc-4.5.0.jar in Lucene binary distribution Should fail: Artifact WEB-INF/lib/lucene-queries-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/queries/lucene-queries-4.5.0.jar in Lucene binary distribution Should fail: Artifact WEB-INF/lib/lucene-queryparser-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/queryparser/lucene-queryparser-4.5.0.jar in Lucene binary distribution Should fail: Artifact WEB-INF/lib/lucene-spatial-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/spatial/lucene-spatial-4.5.0.jar in Lucene binary distribution Should fail: Artifact WEB-INF/lib/lucene-suggest-4.5.0.jar in /Users/sarowe/temp/tmpDir/unpack/solr-4.5.0/example/webapps/solr.war is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/suggest/lucene-suggest-4.5.0.jar in Lucene binary distribution [...] Test Maven artifacts for Lucene and Solr... [...] verify that Maven artifacts are same as in the binary distribution... Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-analyzers-common/4.5.0/lucene-analyzers-common-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/common/lucene-analyzers-common-4.5.0.jar in lucene binary distribution Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-analyzers-icu/4.5.0/lucene-analyzers-icu-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/icu/lucene-analyzers-icu-4.5.0.jar in lucene binary distribution Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-analyzers-kuromoji/4.5.0/lucene-analyzers-kuromoji-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/kuromoji/lucene-analyzers-kuromoji-4.5.0.jar in lucene binary distribution Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-analyzers-morfologik/4.5.0/lucene-analyzers-morfologik-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/morfologik/lucene-analyzers-morfologik-4.5.0.jar in lucene binary distribution Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-analyzers-phonetic/4.5.0/lucene-analyzers-phonetic-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/phonetic/lucene-analyzers-phonetic-4.5.0.jar in lucene binary distribution Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-analyzers-smartcn/4.5.0/lucene-analyzers-smartcn-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/smartcn/lucene-analyzers-smartcn-4.5.0.jar in lucene binary distribution Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-analyzers-stempel/4.5.0/lucene-analyzers-stempel-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/stempel/lucene-analyzers-stempel-4.5.0.jar in lucene binary distribution Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-analyzers-uima/4.5.0/lucene-analyzers-uima-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/analysis/uima/lucene-analyzers-uima-4.5.0.jar in lucene binary distribution Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-benchmark/4.5.0/lucene-benchmark-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/benchmark/lucene-benchmark-4.5.0.jar in lucene binary distribution Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-classification/4.5.0/lucene-classification-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/classification/lucene-classification-4.5.0.jar in lucene binary distribution Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-codecs/4.5.0/lucene-codecs-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/codecs/lucene-codecs-4.5.0.jar in lucene binary distribution Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-core/4.5.0/lucene-core-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/core/lucene-core-4.5.0.jar in lucene binary distribution Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-demo/4.5.0/lucene-demo-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/demo/lucene-demo-4.5.0.jar in lucene binary distribution Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-facet/4.5.0/lucene-facet-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/facet/lucene-facet-4.5.0.jar in lucene binary distribution Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-grouping/4.5.0/lucene-grouping-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/grouping/lucene-grouping-4.5.0.jar in lucene binary distribution Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-highlighter/4.5.0/lucene-highlighter-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/highlighter/lucene-highlighter-4.5.0.jar in lucene binary distribution Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-join/4.5.0/lucene-join-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/join/lucene-join-4.5.0.jar in lucene binary distribution Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-memory/4.5.0/lucene-memory-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/memory/lucene-memory-4.5.0.jar in lucene binary distribution Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-misc/4.5.0/lucene-misc-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/misc/lucene-misc-4.5.0.jar in lucene binary distribution Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-queries/4.5.0/lucene-queries-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/queries/lucene-queries-4.5.0.jar in lucene binary distribution Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-queryparser/4.5.0/lucene-queryparser-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/queryparser/lucene-queryparser-4.5.0.jar in lucene binary distribution Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-replicator/4.5.0/lucene-replicator-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/replicator/lucene-replicator-4.5.0.jar in lucene binary distribution Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-sandbox/4.5.0/lucene-sandbox-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/sandbox/lucene-sandbox-4.5.0.jar in lucene binary distribution Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-spatial/4.5.0/lucene-spatial-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/spatial/lucene-spatial-4.5.0.jar in lucene binary distribution Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-suggest/4.5.0/lucene-suggest-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/suggest/lucene-suggest-4.5.0.jar in lucene binary distribution Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/lucene/lucene-test-framework/4.5.0/lucene-test-framework-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-lucene-getBinaryDistFiles/lucene-4.5.0/test-framework/lucene-test-framework-4.5.0.jar in lucene binary distribution Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/solr/solr-analysis-extras/4.5.0/solr-analysis-extras-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-solr-getBinaryDistFiles/solr-4.5.0/dist/solr-analysis-extras-4.5.0.jar in solr binary distribution Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/solr/solr-cell/4.5.0/solr-cell-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-solr-getBinaryDistFiles/solr-4.5.0/dist/solr-cell-4.5.0.jar in solr binary distribution Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/solr/solr-clustering/4.5.0/solr-clustering-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-solr-getBinaryDistFiles/solr-4.5.0/dist/solr-clustering-4.5.0.jar in solr binary distribution Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/solr/solr-core/4.5.0/solr-core-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-solr-getBinaryDistFiles/solr-4.5.0/dist/solr-core-4.5.0.jar in solr binary distribution Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/solr/solr-dataimporthandler-extras/4.5.0/solr-dataimporthandler-extras-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-solr-getBinaryDistFiles/solr-4.5.0/dist/solr-dataimporthandler-extras-4.5.0.jar in solr binary distribution Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/solr/solr-dataimporthandler/4.5.0/solr-dataimporthandler-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-solr-getBinaryDistFiles/solr-4.5.0/dist/solr-dataimporthandler-4.5.0.jar in solr binary distribution Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/solr/solr-langid/4.5.0/solr-langid-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-solr-getBinaryDistFiles/solr-4.5.0/dist/solr-langid-4.5.0.jar in solr binary distribution Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/solr/solr-solrj/4.5.0/solr-solrj-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-solr-getBinaryDistFiles/solr-4.5.0/dist/solr-solrj-4.5.0.jar in solr binary distribution Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/solr/solr-test-framework/4.5.0/solr-test-framework-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-solr-getBinaryDistFiles/solr-4.5.0/dist/solr-test-framework-4.5.0.jar in solr binary distribution Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/solr/solr-uima/4.5.0/solr-uima-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-solr-getBinaryDistFiles/solr-4.5.0/dist/solr-uima-4.5.0.jar in solr binary distribution Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/solr/solr-velocity/4.5.0/solr-velocity-4.5.0.jar is not identical to /Users/sarowe/temp/tmpDir/unpack-solr-getBinaryDistFiles/solr-4.5.0/dist/solr-velocity-4.5.0.jar in solr binary distribution Should fail: Maven artifact /Users/sarowe/temp/tmpDir/maven/org/apache/solr/solr/4.5.0/solr-4.5.0.war is not identical to /Users/sarowe/temp/tmpDir/unpack-solr-getBinaryDistFiles/solr-4.5.0/dist/solr-4.5.0.war in solr binary distribution So I think it works.
        Hide
        Uwe Schindler added a comment -

        Just one question: Why unpack the whole lucene tgz? You can directly refer to JAR files from inside tar/zip by using zipfileset/tarfileset. So there is no need to unpack those files, just add them directly from their tar resource. I use a similar approach in forbidden, to extract stuff from IVY artifacts without copying and unzipping JAR files.

        Show
        Uwe Schindler added a comment - Just one question: Why unpack the whole lucene tgz? You can directly refer to JAR files from inside tar/zip by using zipfileset/tarfileset. So there is no need to unpack those files, just add them directly from their tar resource. I use a similar approach in forbidden, to extract stuff from IVY artifacts without copying and unzipping JAR files.
        Hide
        Uwe Schindler added a comment - - edited

        Here my example from forbidden: https://code.google.com/p/forbidden-apis/source/browse/trunk/build.xml?spec=svn214&r=214#215
        It uses <archives><zips/></archives> when zipping the artifact. The same can be done with <archives><tars/></archives>. Another way to do this (more static) is to use <tarfileset/>. <archives/> is just more flexible as it is a generic resource collection.

        Show
        Uwe Schindler added a comment - - edited Here my example from forbidden: https://code.google.com/p/forbidden-apis/source/browse/trunk/build.xml?spec=svn214&r=214#215 It uses <archives><zips/></archives> when zipping the artifact. The same can be done with <archives><tars/></archives>. Another way to do this (more static) is to use <tarfileset/>. <archives/> is just more flexible as it is a generic resource collection.
        Hide
        Steve Rowe added a comment -

        Good idea Uwe, thanks for the pointers, I'll look at that later on today.

        Converting dependency paths referring to jars under lucene/build/ and solr/build/ to the corresponding files in the unpacked binary distribution took some doing... hopefully that'll also be possible with <tarfileset/>.

        Show
        Steve Rowe added a comment - Good idea Uwe, thanks for the pointers, I'll look at that later on today. Converting dependency paths referring to jars under lucene/build/ and solr/build/ to the corresponding files in the unpacked binary distribution took some doing... hopefully that'll also be possible with <tarfileset/> .
        Hide
        Steve Rowe added a comment -

        Hmm, the Maven Ant Tasks install and deploy tasks require that the source resources be files - at least, the documentation makes no mention of anything but files - so I don't think there's any way to avoid unpacking the archives.

        Show
        Steve Rowe added a comment - Hmm, the Maven Ant Tasks install and deploy tasks require that the source resources be files - at least, the documentation makes no mention of anything but files - so I don't think there's any way to avoid unpacking the archives.
        Hide
        Robert Muir added a comment -

        strong +1, i always hated how we release different jars in different places. the current setup is just asking for trouble.

        Show
        Robert Muir added a comment - strong +1, i always hated how we release different jars in different places. the current setup is just asking for trouble.
        Hide
        Steve Rowe added a comment -

        I'll commit this as-is tomorrow if there are no objections.

        Show
        Steve Rowe added a comment - I'll commit this as-is tomorrow if there are no objections.
        Hide
        Mark Miller added a comment -

        +1

        Show
        Mark Miller added a comment - +1
        Hide
        ASF subversion and git services added a comment -

        Commit 1531354 from Steve Rowe in branch 'dev/trunk'
        [ https://svn.apache.org/r1531354 ]

        LUCENE-5273: Binary artifacts in Lucene and Solr convenience binary distributions accompanying a release, including on Maven Central, should be identical across all distributions.

        Show
        ASF subversion and git services added a comment - Commit 1531354 from Steve Rowe in branch 'dev/trunk' [ https://svn.apache.org/r1531354 ] LUCENE-5273 : Binary artifacts in Lucene and Solr convenience binary distributions accompanying a release, including on Maven Central, should be identical across all distributions.
        Hide
        ASF subversion and git services added a comment -

        Commit 1531507 from Steve Rowe in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1531507 ]

        LUCENE-5273: Binary artifacts in Lucene and Solr convenience binary distributions accompanying a release, including on Maven Central, should be identical across all distributions. (merged trunk r1531354)

        Show
        ASF subversion and git services added a comment - Commit 1531507 from Steve Rowe in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1531507 ] LUCENE-5273 : Binary artifacts in Lucene and Solr convenience binary distributions accompanying a release, including on Maven Central, should be identical across all distributions. (merged trunk r1531354)
        Hide
        Steve Rowe added a comment -

        Mark Miller, do you want this on the 4.5 release branch? It would produce a better set of distributions, but is maybe a little too risky, so soon after committing it?

        FWIW, ant nightly-smoke passed for me on both trunk and branch_4x before I committed each place, and I re-ran the 4.5RC5 test from branch_4x (converting exceptions to debug prints for the non-identicality tests and for missing stuff, to allow the script to continue e.g. when it didn't find the new expressions module), and it complained in the right places.

        Show
        Steve Rowe added a comment - Mark Miller , do you want this on the 4.5 release branch? It would produce a better set of distributions, but is maybe a little too risky, so soon after committing it? FWIW, ant nightly-smoke passed for me on both trunk and branch_4x before I committed each place, and I re-ran the 4.5RC5 test from branch_4x (converting exceptions to debug prints for the non-identicality tests and for missing stuff, to allow the script to continue e.g. when it didn't find the new expressions module), and it complained in the right places.
        Hide
        Uwe Schindler added a comment - - edited

        Hmm, the Maven Ant Tasks install and deploy tasks require that the source resources be files - at least, the documentation makes no mention of anything but files - so I don't think there's any way to avoid unpacking the archives.

        Yeah, I noticed this, too.

        I am not sure of the overhead, but as the binary artifact also contains javadocs and those are many small files, this makes the task run slowly on windows and some filesystems. How about only extracting JAR files on the untar task? I would only add some patternset matching only **/*.jar to the untar task (see last example on http://ant.apache.org/manual/Tasks/unzip.html).

        I will try this out soon, just need breakfast!

        Show
        Uwe Schindler added a comment - - edited Hmm, the Maven Ant Tasks install and deploy tasks require that the source resources be files - at least, the documentation makes no mention of anything but files - so I don't think there's any way to avoid unpacking the archives. Yeah, I noticed this, too. I am not sure of the overhead, but as the binary artifact also contains javadocs and those are many small files, this makes the task run slowly on windows and some filesystems. How about only extracting JAR files on the untar task? I would only add some patternset matching only **/*.jar to the untar task (see last example on http://ant.apache.org/manual/Tasks/unzip.html ). I will try this out soon, just need breakfast!
        Hide
        Uwe Schindler added a comment -

        Here the patch. ant generate-maven-articfatcs succeeds (it would fail if some needed files were not extracted from tgz). I did not smoke, as I have only windows.

        In general there is one problem with the whole approach of untaring: It seems to me that you can no longer build maven artifacts for a single module, it still builds everything and fails lately (e.g. dist-maven). So I would suggest to rename those tasks to use "-" as prefix, so you can no longer call them: "dist-maven" is one that is affected, rename to "-dist-maven". We should do this in a separate issue.

        Steve Rowe: Should I commit this optimization? It also largely reduces disk usage!

        Show
        Uwe Schindler added a comment - Here the patch. ant generate-maven-articfatcs succeeds (it would fail if some needed files were not extracted from tgz). I did not smoke, as I have only windows. In general there is one problem with the whole approach of untaring: It seems to me that you can no longer build maven artifacts for a single module, it still builds everything and fails lately (e.g. dist-maven). So I would suggest to rename those tasks to use "-" as prefix, so you can no longer call them: "dist-maven" is one that is affected, rename to "-dist-maven". We should do this in a separate issue. Steve Rowe : Should I commit this optimization? It also largely reduces disk usage!
        Hide
        Uwe Schindler added a comment -

        BTW: I was hoping this patch also prevent rebuilding the same javadocs over and over We should also fix this in the future, it's horrible. While building Solr maven artifacts it rebuilds solr-core javadocs 5 or 6 times! (for each module),

        Steve Rowe: I am out of office, you can commit this patch and also backport if you like!

        Show
        Uwe Schindler added a comment - BTW: I was hoping this patch also prevent rebuilding the same javadocs over and over We should also fix this in the future, it's horrible. While building Solr maven artifacts it rebuilds solr-core javadocs 5 or 6 times! (for each module), Steve Rowe : I am out of office, you can commit this patch and also backport if you like!
        Hide
        Shalin Shekhar Mangar added a comment -

        I just noticed on trunk that executing "ant example" for solr causes lucene tgz file (and hence javadocs) to be built. So the "example" task takes quite a bit longer to execute:

        cd solr; ant example
        branches/lucene_solr_4_5: Total time: 1 minute 33 seconds
        trunk: Total time: 4 minutes 54 seconds

        Show
        Shalin Shekhar Mangar added a comment - I just noticed on trunk that executing "ant example" for solr causes lucene tgz file (and hence javadocs) to be built. So the "example" task takes quite a bit longer to execute: cd solr; ant example branches/lucene_solr_4_5: Total time: 1 minute 33 seconds trunk: Total time: 4 minutes 54 seconds
        Hide
        Steve Rowe added a comment -

        Thanks for reporting the slowdown, Shalin, I'll look into it.

        Show
        Steve Rowe added a comment - Thanks for reporting the slowdown, Shalin, I'll look into it.
        Hide
        ASF subversion and git services added a comment -

        Commit 1531711 from Steve Rowe in branch 'dev/trunk'
        [ https://svn.apache.org/r1531711 ]

        LUCENE-5273: Only unpack Lucene/Solr jars from binary distributions - other stuff not needed

        Show
        ASF subversion and git services added a comment - Commit 1531711 from Steve Rowe in branch 'dev/trunk' [ https://svn.apache.org/r1531711 ] LUCENE-5273 : Only unpack Lucene/Solr jars from binary distributions - other stuff not needed
        Hide
        ASF subversion and git services added a comment -

        Commit 1531724 from Steve Rowe in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1531724 ]

        LUCENE-5273: Only unpack Lucene/Solr jars/war from binary distributions - other stuff not needed (merged trunk r1531711)

        Show
        ASF subversion and git services added a comment - Commit 1531724 from Steve Rowe in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1531724 ] LUCENE-5273 : Only unpack Lucene/Solr jars/war from binary distributions - other stuff not needed (merged trunk r1531711)
        Hide
        Steve Rowe added a comment -

        I am not sure of the overhead, but as the binary artifact also contains javadocs and those are many small files, this makes the task run slowly on windows and some filesystems. How about only extracting JAR files on the untar task? I would only add some patternset matching only */.jar to the untar task (see last example on http://ant.apache.org/manual/Tasks/unzip.html).
        [...]
        Steve Rowe: I am out of office, you can commit this patch and also backport if you like!

        Thanks Uwe, I committed (with one small change to the patternset name) to trunk and branch_4x. I added the Solr war to the patternset-to-unpack on branch_4x. ant nightly-smoke passed for me in both places.

        Show
        Steve Rowe added a comment - I am not sure of the overhead, but as the binary artifact also contains javadocs and those are many small files, this makes the task run slowly on windows and some filesystems. How about only extracting JAR files on the untar task? I would only add some patternset matching only * / .jar to the untar task (see last example on http://ant.apache.org/manual/Tasks/unzip.html ). [...] Steve Rowe: I am out of office, you can commit this patch and also backport if you like! Thanks Uwe, I committed (with one small change to the patternset name) to trunk and branch_4x. I added the Solr war to the patternset-to-unpack on branch_4x. ant nightly-smoke passed for me in both places.
        Hide
        Steve Rowe added a comment - - edited

        Patch addressing the slowdown Shalin mentioned above: the old war-building behavior of pulling lucene-*.jar from lucene/build/*/ is restored, unless called from the create-package target, in which case they are pulled from the built/unpacked lucene-<version>.tgz. In this way, ant example doesn't have to wait around for javadocs.

        Here are timings from my Macbook Pro (OS X 10.8.5) with Oracle Java 1.7.0_25 64-bit - times are minutes:seconds:

        Top-level clean? trunk before any LUCENE-5273 patches committed (r1531353) current trunk trunk with this patch applied
        Yes 1:11 4:26 1:15
        No 0:37 0:38 0:37

        So with this patch, timings are back to where they were before I committed anything for this issue.

        This patch provides the additional benefit that Lucene module changes will get picked up even if ant clean is not performed first.

        Committing shortly.

        Show
        Steve Rowe added a comment - - edited Patch addressing the slowdown Shalin mentioned above: the old war-building behavior of pulling lucene-*.jar from lucene/build/*/ is restored, unless called from the create-package target, in which case they are pulled from the built/unpacked lucene-<version>.tgz . In this way, ant example doesn't have to wait around for javadocs. Here are timings from my Macbook Pro (OS X 10.8.5) with Oracle Java 1.7.0_25 64-bit - times are minutes:seconds : Top-level clean? trunk before any LUCENE-5273 patches committed (r1531353) current trunk trunk with this patch applied Yes 1:11 4:26 1:15 No 0:37 0:38 0:37 So with this patch, timings are back to where they were before I committed anything for this issue. This patch provides the additional benefit that Lucene module changes will get picked up even if ant clean is not performed first. Committing shortly.
        Hide
        Robert Muir added a comment -

        BTW: I was hoping this patch also prevent rebuilding the same javadocs over and over We should also fix this in the future, it's horrible. While building Solr maven artifacts it rebuilds solr-core javadocs 5 or 6 times! (for each module),

        I think we should open a followup issue for this? There is definitely something crazy going on here, it adds a considerable amount of time to tasks like precommit, documentation-lint, release packaging, smoke testing, etc.

        Show
        Robert Muir added a comment - BTW: I was hoping this patch also prevent rebuilding the same javadocs over and over We should also fix this in the future, it's horrible. While building Solr maven artifacts it rebuilds solr-core javadocs 5 or 6 times! (for each module), I think we should open a followup issue for this? There is definitely something crazy going on here, it adds a considerable amount of time to tasks like precommit, documentation-lint, release packaging, smoke testing, etc.
        Hide
        ASF subversion and git services added a comment -

        Commit 1531750 from Steve Rowe in branch 'dev/trunk'
        [ https://svn.apache.org/r1531750 ]

        LUCENE-5273: Fix slowdown when running 'ant example': unless called from the 'create-package' target, the 'lucene-jars-to-solr' and 'module-jars-to-solr' targets no longer depend on 'unpack-lucene-tgz', pulling lucene.jar from lucene/build// instead.

        Show
        ASF subversion and git services added a comment - Commit 1531750 from Steve Rowe in branch 'dev/trunk' [ https://svn.apache.org/r1531750 ] LUCENE-5273 : Fix slowdown when running 'ant example': unless called from the 'create-package' target, the 'lucene-jars-to-solr' and 'module-jars-to-solr' targets no longer depend on ' unpack-lucene-tgz', pulling lucene .jar from lucene/build/ / instead.
        Hide
        Mark Miller added a comment -

        Mark Miller, do you want this on the 4.5 release branch?

        I'll defer to those of you deeper into the issue, but from a high level, to me it does not seem worth the gain vs the risk for 4.5.1.

        Show
        Mark Miller added a comment - Mark Miller, do you want this on the 4.5 release branch? I'll defer to those of you deeper into the issue, but from a high level, to me it does not seem worth the gain vs the risk for 4.5.1.
        Hide
        Robert Muir added a comment -

        I agree with Mark: as much as I like the change, and even though I consider it a "bug", I think its best to be conservative and to just fix this for 4.6?

        Show
        Robert Muir added a comment - I agree with Mark: as much as I like the change, and even though I consider it a "bug", I think its best to be conservative and to just fix this for 4.6?
        Hide
        ASF subversion and git services added a comment -

        Commit 1531759 from Steve Rowe in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1531759 ]

        LUCENE-5273: Fix slowdown when running 'ant example': unless called from the 'create-package' target, the 'lucene-jars-to-solr' and 'module-jars-to-solr' targets no longer depend on 'unpack-lucene-tgz', pulling lucene.jar from lucene/build// instead. (merged trunk r1531750)

        Show
        ASF subversion and git services added a comment - Commit 1531759 from Steve Rowe in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1531759 ] LUCENE-5273 : Fix slowdown when running 'ant example': unless called from the 'create-package' target, the 'lucene-jars-to-solr' and 'module-jars-to-solr' targets no longer depend on ' unpack-lucene-tgz', pulling lucene .jar from lucene/build/ / instead. (merged trunk r1531750)
        Hide
        Steve Rowe added a comment -

        Slowdown fix committed to trunk and branch_4x.

        I won't backport any of this to the lucene_solr_4_5 branch.

        Thanks guys!

        Show
        Steve Rowe added a comment - Slowdown fix committed to trunk and branch_4x. I won't backport any of this to the lucene_solr_4_5 branch. Thanks guys!

          People

          • Assignee:
            Steve Rowe
            Reporter:
            Steve Rowe
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development