Lucene - Core
  1. Lucene - Core
  2. LUCENE-2326

Remove SVN.exe and revision numbers from build.xml by svn-copy the backwards branch and linking snowball tests by svn:externals

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.0-ALPHA
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New, Patch Available

      Description

      As we often need to update backwards tests together with trunk and always have to update the branch first, record rev no, and update build xml, I would simply like to do a svn copy/move of the backwards branch.

      After a release, this is simply also done:

      svn rm backwards
      svn cp releasebranch backwards
      

      By this we can simply commit in one pass, create patches in one pass.

      The snowball tests are currently downloaded by svn.exe, too. These need a fixed version for checkout. I would like to change this to use svn:externals. Will provide patch, soon.

      1. LUCENE-2326.patch
        13 kB
        Uwe Schindler
      2. LUCENE-2326.patch
        11 kB
        Uwe Schindler
      3. LUCENE-2326-snowball-try2.patch
        5 kB
        Uwe Schindler
      4. TestVnowballVocabData.zip
        2.98 MB
        Uwe Schindler

        Activity

        Hide
        Robert Muir added a comment -

        I agree i think its nice to see a patch to lucene that includes
        any changes to the backwards tests.

        Mike did this with LUCENE-2111 and i was shocked, until
        I found out he was doing it manually with cat.

        Show
        Robert Muir added a comment - I agree i think its nice to see a patch to lucene that includes any changes to the backwards tests. Mike did this with LUCENE-2111 and i was shocked, until I found out he was doing it manually with cat.
        Hide
        Uwe Schindler added a comment -

        I think the ideal case for this would be that the backwards folder simply contains the src-folder of the previous branch (after creation). No extra folder like now in between, so it looks like "/backwards/src/...". After a release, one would first "svn rm" the old and then "svn copy" the src folder of the previously created release branch to trunk. I would add this to the release todo.

        On this change, all committers must first manually do a operating-system "rm -rf" on the backwards folder by calling "ant clean-backwards" before svn up. Maybe create a patch before

        Show
        Uwe Schindler added a comment - I think the ideal case for this would be that the backwards folder simply contains the src-folder of the previous branch (after creation). No extra folder like now in between, so it looks like "/backwards/src/...". After a release, one would first "svn rm" the old and then "svn copy" the src folder of the previously created release branch to trunk. I would add this to the release todo. On this change, all committers must first manually do a operating-system "rm -rf" on the backwards folder by calling "ant clean-backwards" before svn up. Maybe create a patch before
        Hide
        Michael McCandless added a comment -

        +1

        This sounds sooo much better than what we do now.

        Show
        Michael McCandless added a comment - +1 This sounds sooo much better than what we do now.
        Hide
        Uwe Schindler added a comment - - edited

        Here the patch, before applying do the following (in main checkout folder):

        ant clean-backwards
        svn mkdir ./backwards
        svn cp https://svn.apache.org/repos/asf/lucene/java/branches/lucene_3_0_back_compat_tests/src backwards/src
        svn propset svn:externals "data -r500 svn://svn.tartarus.org/snowball/trunk/data" contrib/analyzers/common/src/test/org/apache/lucene/analysis/snowball
        svn propdel svn:ignore contrib/analyzers/common/src/test/org/apache/lucene/analysis/snowball
        

        Then apply patch and run svn up.

        Show
        Uwe Schindler added a comment - - edited Here the patch, before applying do the following (in main checkout folder): ant clean-backwards svn mkdir ./backwards svn cp https://svn.apache.org/repos/asf/lucene/java/branches/lucene_3_0_back_compat_tests/src backwards/src svn propset svn:externals "data -r500 svn://svn.tartarus.org/snowball/trunk/data" contrib/analyzers/common/src/test/org/apache/lucene/analysis/snowball svn propdel svn:ignore contrib/analyzers/common/src/test/org/apache/lucene/analysis/snowball Then apply patch and run svn up.
        Hide
        Uwe Schindler added a comment -

        I added one thing (as discussed with rmuir):
        As the snowball test data is too much, i excluded it from the src jar. The test will not fail, but instead print a warning, that the data is missing. So the test will also pass, if e.g. hudson fails to checkout the external svn repo.

        Show
        Uwe Schindler added a comment - I added one thing (as discussed with rmuir): As the snowball test data is too much, i excluded it from the src jar. The test will not fail, but instead print a warning, that the data is missing. So the test will also pass, if e.g. hudson fails to checkout the external svn repo.
        Hide
        Uwe Schindler added a comment -

        New patch, which has some optimizations. It now also allows to run "ant test" from a source distribution ZIP/TGZ, which does not contain the backwards folder. The tests will not fail, instead print a warning message.

        Show
        Uwe Schindler added a comment - New patch, which has some optimizations. It now also allows to run "ant test" from a source distribution ZIP/TGZ, which does not contain the backwards folder. The tests will not fail, instead print a warning message.
        Hide
        Uwe Schindler added a comment -

        Will commit soon to trunk and merge to flex.

        Show
        Uwe Schindler added a comment - Will commit soon to trunk and merge to flex.
        Hide
        Uwe Schindler added a comment -

        Committed revision: 924207

        Show
        Uwe Schindler added a comment - Committed revision: 924207
        Hide
        Robert Muir added a comment -

        This use of svn:externals causes a problem for snowball, it does not always fetch the correct revision

        [junit] Testcase: testStemmers(org.apache.lucene.analysis.snowball.TestSnowballVocab): FAILED
        [junit] term 0 expected:<amtsgeheimnis[]> but was:<amtsgeheimnis[s]>
        [junit] junit.framework.ComparisonFailure: term 0 expected:<amtsgeheimnis[]> but was:<amtsgeheimnis[s]>

        This is sporatic and not easy to reproduce.

        You can clearly see that its fetching the wrong revision by looking at:
        http://svn.tartarus.org/snowball/trunk/data/german/output.txt?r1=432&r2=527

        Where rev 527 expects "amtsgeheimnis", but rev 500 should expect "amtsgeheimniss"

        Show
        Robert Muir added a comment - This use of svn:externals causes a problem for snowball, it does not always fetch the correct revision [junit] Testcase: testStemmers(org.apache.lucene.analysis.snowball.TestSnowballVocab): FAILED [junit] term 0 expected:<amtsgeheimnis[]> but was:<amtsgeheimnis [s] > [junit] junit.framework.ComparisonFailure: term 0 expected:<amtsgeheimnis[]> but was:<amtsgeheimnis [s] > This is sporatic and not easy to reproduce. You can clearly see that its fetching the wrong revision by looking at: http://svn.tartarus.org/snowball/trunk/data/german/output.txt?r1=432&r2=527 Where rev 527 expects "amtsgeheimnis", but rev 500 should expect "amtsgeheimniss"
        Hide
        Uwe Schindler added a comment -

        What did you to for this to happen?

        You can only reproduce this (and this was also possible with your previous setup), if you go onto the data folder and update there. If you update from top-level (outside the data folder), it works always. Maybe the problem lies in the fact, that you had the data already checked out before our reorganisation (from previous test runs). Can you simply delete the data folder with a OS' rm and update again?

        Maybe it was a problem with svn server?

        Show
        Uwe Schindler added a comment - What did you to for this to happen? You can only reproduce this (and this was also possible with your previous setup), if you go onto the data folder and update there. If you update from top-level (outside the data folder), it works always. Maybe the problem lies in the fact, that you had the data already checked out before our reorganisation (from previous test runs). Can you simply delete the data folder with a OS' rm and update again? Maybe it was a problem with svn server?
        Hide
        Robert Muir added a comment -

        What did you to for this to happen?

        Uwe, the problem happened to Mark... and this test data has always been rev 500.

        svn.exe simply got the wrong revision. Its probably a bug in svn, I don't think you did anything wrong.

        But at the same time, we don't want random test failures.

        Show
        Robert Muir added a comment - What did you to for this to happen? Uwe, the problem happened to Mark... and this test data has always been rev 500. svn.exe simply got the wrong revision. Its probably a bug in svn, I don't think you did anything wrong. But at the same time, we don't want random test failures.
        Hide
        Uwe Schindler added a comment -

        Man, I reverted the snowball part.

        Lets change to a zip file as the tests will never change. This svn in build.xml is too much dependent on your local installation of svn tools. I dont like it.

        Show
        Uwe Schindler added a comment - Man, I reverted the snowball part. Lets change to a zip file as the tests will never change. This svn in build.xml is too much dependent on your local installation of svn tools. I dont like it.
        Hide
        Robert Muir added a comment -

        Lets change to a zip file as the tests will never change

        I agree, but this zip file will be pretty large!

        Thanks for temporarily changing it to do the checkout instead

        Show
        Robert Muir added a comment - Lets change to a zip file as the tests will never change I agree, but this zip file will be pretty large! Thanks for temporarily changing it to do the checkout instead
        Hide
        Uwe Schindler added a comment -

        Here the patch without external references. The data dir was cleaned up (removed the large unneeded diff.txt files) and the zip compressed with -9.

        Show
        Uwe Schindler added a comment - Here the patch without external references. The data dir was cleaned up (removed the large unneeded diff.txt files) and the zip compressed with -9.
        Hide
        Uwe Schindler added a comment -

        Sorry, ZIP file has wrong name. Fixed here locally (test+zip).

        Show
        Uwe Schindler added a comment - Sorry, ZIP file has wrong name. Fixed here locally (test+zip).
        Hide
        Robert Muir added a comment -

        Thanks Uwe, this simplifies our tests.

        Its nice to remove a network connection (it seems reliable so far, but...)

        Show
        Robert Muir added a comment - Thanks Uwe, this simplifies our tests. Its nice to remove a network connection (it seems reliable so far, but...)
        Hide
        Uwe Schindler added a comment -

        Committed revision: 924781 (with correct zip file name)

        Show
        Uwe Schindler added a comment - Committed revision: 924781 (with correct zip file name)

          People

          • Assignee:
            Uwe Schindler
            Reporter:
            Uwe Schindler
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development