Lucene - Core
  1. Lucene - Core
  2. LUCENE-1845

if the build fails to download JARs for contrib/db, just skip its tests

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.0-ALPHA
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Every so often our nightly build fails because contrib/db is unable to download the necessary BDB JARs from http://downloads.osafoundation.org. I think in such cases we should simply skip contrib/db's tests, if it's the nightly build that's running, since it's a false positive failure.

      1. LUCENE-1845.patch
        2 kB
        Simon Willnauer
      2. LUCENE-1845.patch
        4 kB
        Simon Willnauer
      3. LUCENE-1845.txt
        3 kB
        Simon Willnauer
      4. LUCENE-1845.txt
        2 kB
        Simon Willnauer
      5. LUCENE-1845.txt
        2 kB
        Simon Willnauer
      6. LUCENE-1845.txt
        0.6 kB
        Simon Willnauer

        Activity

        Hide
        Simon Willnauer added a comment -

        I set the property "ignoreerrors" to true on the get task. This should print out if there is a problem with the download and continue. The sanity check will fail if the jar is not present and unit-test will be skipped.
        i guess that should do the job though.

        Show
        Simon Willnauer added a comment - I set the property "ignoreerrors" to true on the get task. This should print out if there is a problem with the download and continue. The sanity check will fail if the jar is not present and unit-test will be skipped. i guess that should do the job though.
        Hide
        Michael McCandless added a comment -

        Hmm – I tried applying the patch, then changing the download URL to something bogus that fails, and then "ant test" hits errors during the "compile-core" target. It seems like we have to somehow skip compile-core if the sanity check fails?

        Show
        Michael McCandless added a comment - Hmm – I tried applying the patch, then changing the download URL to something bogus that fails, and then "ant test" hits errors during the "compile-core" target. It seems like we have to somehow skip compile-core if the sanity check fails?
        Hide
        Simon Willnauer added a comment -

        Weird! - I changed the URL to http://foo.bar and ant.test succeeds with the expected message. I guess you changed the get url in bdb-je/build.xml but this file (the je.jar) is not the cause of this issue unless I got something wrong. I thought this issues is caused by the fact that http://downloads.osafoundation.org/db/db-$

        {db.version}

        .jar is not available on a regular basis. Thats why I did not patch this sub-module.

        simon

        Show
        Simon Willnauer added a comment - Weird! - I changed the URL to http://foo.bar and ant.test succeeds with the expected message. I guess you changed the get url in bdb-je/build.xml but this file (the je.jar) is not the cause of this issue unless I got something wrong. I thought this issues is caused by the fact that http://downloads.osafoundation.org/db/db-$ {db.version} .jar is not available on a regular basis. Thats why I did not patch this sub-module. simon
        Hide
        Michael McCandless added a comment -

        Aha, you're right! Sorry about the confusion.

        OK so this is good to go. Can you commit?

        Show
        Michael McCandless added a comment - Aha, you're right! Sorry about the confusion. OK so this is good to go. Can you commit?
        Hide
        Simon Willnauer added a comment -

        OK so this is good to go. Can you commit?

        will do!

        Show
        Simon Willnauer added a comment - OK so this is good to go. Can you commit? will do!
        Hide
        Simon Willnauer added a comment -

        Mike, I attached a new patch. The old one had some problems with the sanity check as the check needs the jar though.
        This one will work for unit-tests but it will fail if ant tries to run compile-core during a build,jar, release etc.

        how should we handle if the jar can not be obtained? I would rather say the build must fail asif we du a release build the jar will not be included.
        Would it be an option to put the jar to some other location maybe on a commiter page on people.apache.org?!

        simon

        Show
        Simon Willnauer added a comment - Mike, I attached a new patch. The old one had some problems with the sanity check as the check needs the jar though. This one will work for unit-tests but it will fail if ant tries to run compile-core during a build,jar, release etc. how should we handle if the jar can not be obtained? I would rather say the build must fail asif we du a release build the jar will not be included. Would it be an option to put the jar to some other location maybe on a commiter page on people.apache.org?! simon
        Hide
        Simon Willnauer added a comment -

        this time with ASF licence grant

        Show
        Simon Willnauer added a comment - this time with ASF licence grant
        Hide
        Michael McCandless added a comment -

        The old one had some problems with the sanity check as the check needs the jar though.

        Ahh OK.

        how should we handle if the jar can not be obtained? I would rather say the build must fail asif we du a release build the jar will not be included.

        I think, ideally, If it's the nightly build that's running, the disposition should be robust (ignored), else it should be brittle (like it is today).

        I'm not sure about hosting the JARs. The license for bdb-je is at http://www.oracle.com/technology/software/products/berkeley-db/htdocs/jeoslicense.html and for bdb http://www.oracle.com/technology/software/products/berkeley-db/htdocs/oslicense.html. I think these libs were LGPL'd before Oracle acquired Sleepycat, which was why we could not host them, but I have no idea if Oracle's licensing would permit us to host/redistribute them. Maybe an inquiry on legal@ is the next step? If the licensing allows it, I think the simplest solution would be to check them in to our svn repository....

        Show
        Michael McCandless added a comment - The old one had some problems with the sanity check as the check needs the jar though. Ahh OK. how should we handle if the jar can not be obtained? I would rather say the build must fail asif we du a release build the jar will not be included. I think, ideally, If it's the nightly build that's running, the disposition should be robust (ignored), else it should be brittle (like it is today). I'm not sure about hosting the JARs. The license for bdb-je is at http://www.oracle.com/technology/software/products/berkeley-db/htdocs/jeoslicense.html and for bdb http://www.oracle.com/technology/software/products/berkeley-db/htdocs/oslicense.html . I think these libs were LGPL'd before Oracle acquired Sleepycat, which was why we could not host them, but I have no idea if Oracle's licensing would permit us to host/redistribute them. Maybe an inquiry on legal@ is the next step? If the licensing allows it, I think the simplest solution would be to check them in to our svn repository....
        Hide
        Simon Willnauer added a comment -

        Checking it into svn would be an option as far as I know just having something in SVN would not violate licenses as we do not distribute it but I will check with legal.

        simon

        Show
        Simon Willnauer added a comment - Checking it into svn would be an option as far as I know just having something in SVN would not violate licenses as we do not distribute it but I will check with legal. simon
        Hide
        Michael McCandless added a comment -

        Last 3 builds were broken because of this... can we "improve" the sanity checking to check for lack of existence of the JAR, and then skip everything?

        Show
        Michael McCandless added a comment - Last 3 builds were broken because of this... can we "improve" the sanity checking to check for lack of existence of the JAR, and then skip everything?
        Hide
        Simon Willnauer added a comment -

        The current patch disables every build (jar, compile-core, compile-test) if the jar can not be downloaded. It will print a message in the following cases:

        • if the file can not be downloaded
        • if the sanity check does not succeed

        @mike could you have a look at it and try to run it with the <get> commented out as the jar is now available again :/

        simon

        Show
        Simon Willnauer added a comment - The current patch disables every build (jar, compile-core, compile-test) if the jar can not be downloaded. It will print a message in the following cases: if the file can not be downloaded if the sanity check does not succeed @mike could you have a look at it and try to run it with the <get> commented out as the jar is now available again :/ simon
        Hide
        Michael McCandless added a comment -

        Looks great Simon! I removed the JAR, changed the <get> to a bogus host, and confirmed that it did not fail.

        Now the only question is how to not fail only if it's the nightly build. Note that the nightly build runs "ant nightly", so can we somehow set something under the "nightly" target, that contrib/db/bdb's build.xml could access, to know it's the nightly build?

        Show
        Michael McCandless added a comment - Looks great Simon! I removed the JAR, changed the <get> to a bogus host, and confirmed that it did not fail. Now the only question is how to not fail only if it's the nightly build. Note that the nightly build runs "ant nightly", so can we somehow set something under the "nightly" target, that contrib/db/bdb's build.xml could access, to know it's the nightly build?
        Hide
        Simon Willnauer added a comment -

        I was not aware of the "nightly" target. Thanks for pointing it out.

        I added a "nightly" property to trunk/build.xml and let the jar-check always succeed if the nightly property is NOT set. I also changed the name of the property from "db-jar-check-success" to "execute-build".

        I can not commit /trunk/build.xml can you do once I have the bdb/build.xml in?

        simon

        Show
        Simon Willnauer added a comment - I was not aware of the "nightly" target. Thanks for pointing it out. I added a "nightly" property to trunk/build.xml and let the jar-check always succeed if the nightly property is NOT set. I also changed the name of the property from "db-jar-check-success" to "execute-build". I can not commit /trunk/build.xml can you do once I have the bdb/build.xml in? simon
        Hide
        Michael McCandless added a comment -

        For some reason I get hunk #3 failing when I try to apply the patch:

        Hunk #3 FAILED at 66.
        1 out of 3 hunks FAILED -- saving rejects to file contrib/db/bdb/build.xml.rej
        

        But the approach sounds good. So if "ant nightly" is the target, and it fails to download the JAR, it'll just warn and continue. But if it's any other target, it'll fail.

        OK, I'll commit the toplevel build.xml changes shortly. They are fully independent (just sets the "nightly" property).

        Show
        Michael McCandless added a comment - For some reason I get hunk #3 failing when I try to apply the patch: Hunk #3 FAILED at 66. 1 out of 3 hunks FAILED -- saving rejects to file contrib/db/bdb/build.xml.rej But the approach sounds good. So if "ant nightly" is the target, and it fails to download the JAR, it'll just warn and continue. But if it's any other target, it'll fail. OK, I'll commit the toplevel build.xml changes shortly. They are fully independent (just sets the "nightly" property).
        Hide
        Michael McCandless added a comment -

        OK I committed the top-level build.xml. Thanks Simon!

        Show
        Michael McCandless added a comment - OK I committed the top-level build.xml. Thanks Simon!
        Hide
        Simon Willnauer added a comment -

        For some reason I get hunk #3 failing when I try to apply the patch:

        Hmm... works for me on a clean checkout. did you revert the last changes?!

        I plan to commit shortly - if anybody does not feel comfortable with it speak up.

        Show
        Simon Willnauer added a comment - For some reason I get hunk #3 failing when I try to apply the patch: Hmm... works for me on a clean checkout. did you revert the last changes?! I plan to commit shortly - if anybody does not feel comfortable with it speak up.
        Hide
        Michael McCandless added a comment -

        Reopening based on java-dev discussion...

        Show
        Michael McCandless added a comment - Reopening based on java-dev discussion...
        Hide
        Simon Willnauer added a comment -

        I added the short discussion I had on legal-discuss for the record.
        One person confirmed that we could add the jar to SVN if we do not redistribute them. I'm not a license guy but I guess we should first figure out what license this particular jar has. It is not a download from the oracle page (you can only get the sources for this particular jar not the binary as a download) but something from http://downloads.osafoundation.org/db/ without any license notice.

        I would suggest to try to build the jar from source with the latest release on the oracle page so we can be sure about the license. Once I have done this I will send another request to legal to confirm tha we are not violating anything.

        The discussion from legal-discuss

        > Hey there,
        > We (lucene) have a contrib project that provides a Index-Directory
        > implementation based on BerkleyDB. This code downloads a jar file from
        > http://downloads.osafoundation.org/... to build and test the code.
        > This jar-file is not included in any distribution and we do not plan
        > to do so. The problem is that the download site is down very
        > frequently so we are looking for another way to obtain the jar. Here
        > is the question do we violate the license if we add the jar-file to
        > the svn repository but not distributing it at all? Another way would
        > be to add the jar to a commiter page on people.apache.org and download
        > it from there.
        > The license is here:
        > http://www.oracle.com/technology/software/products/berkeley-db/htdocs/oslicense.html
        
        Complicated matter.
        BDB seems viral in that anything that uses must be made available in
        source form. So, ASF has no problem fulfilling that requirement, but
        downstream users may. OTOH, you say that the BDB is only used to build
        (do you really need it to build?) and test your implementation, BUT
        you say that you have an implementation based on BDB, so I presume
        that it requires it to run.
        
        My interpretation is;
         * IFF your component is purely optional, having a dependency on BDB
        is Ok, provided it is not shipped with the release and that the user
        is provided with the information that the BDB needs to be downloaded
        separately and advised to review their license.
        
        For your second part; Can you stick the BDB jar(s) somewhere more
        reliably available?
         * Yes, I think so. The license allows distribution in any form,
        source or binary... So, I suggest that you upload it to a dependable
        host, such as SF, ibiblio.org or similar. people.apache.org --> I
        wouldn't recommend it. ASF SVN --> yes, that should be Ok, but there
        is a strong recommendation of not putting JARs in there... Also there
        is a risk that the encumbrance around BDB is forgotten and used beyond
        what is acceptable if it is 'laying around'.
        
        
        Cheer
        
        Show
        Simon Willnauer added a comment - I added the short discussion I had on legal-discuss for the record. One person confirmed that we could add the jar to SVN if we do not redistribute them. I'm not a license guy but I guess we should first figure out what license this particular jar has. It is not a download from the oracle page (you can only get the sources for this particular jar not the binary as a download) but something from http://downloads.osafoundation.org/db/ without any license notice. I would suggest to try to build the jar from source with the latest release on the oracle page so we can be sure about the license. Once I have done this I will send another request to legal to confirm tha we are not violating anything. The discussion from legal-discuss > Hey there, > We (lucene) have a contrib project that provides a Index-Directory > implementation based on BerkleyDB. This code downloads a jar file from > http://downloads.osafoundation.org/... to build and test the code. > This jar-file is not included in any distribution and we do not plan > to do so. The problem is that the download site is down very > frequently so we are looking for another way to obtain the jar. Here > is the question do we violate the license if we add the jar-file to > the svn repository but not distributing it at all? Another way would > be to add the jar to a commiter page on people.apache.org and download > it from there. > The license is here: > http://www.oracle.com/technology/software/products/berkeley-db/htdocs/oslicense.html Complicated matter. BDB seems viral in that anything that uses must be made available in source form. So, ASF has no problem fulfilling that requirement, but downstream users may. OTOH, you say that the BDB is only used to build (do you really need it to build?) and test your implementation, BUT you say that you have an implementation based on BDB, so I presume that it requires it to run. My interpretation is; * IFF your component is purely optional, having a dependency on BDB is Ok, provided it is not shipped with the release and that the user is provided with the information that the BDB needs to be downloaded separately and advised to review their license. For your second part; Can you stick the BDB jar(s) somewhere more reliably available? * Yes, I think so. The license allows distribution in any form, source or binary... So, I suggest that you upload it to a dependable host, such as SF, ibiblio.org or similar. people.apache.org --> I wouldn't recommend it. ASF SVN --> yes, that should be Ok, but there is a strong recommendation of not putting JARs in there... Also there is a risk that the encumbrance around BDB is forgotten and used beyond what is acceptable if it is 'laying around'. Cheer
        Hide
        Michael McCandless added a comment -

        OK I think we should simply revert our changes to the build.xml's, now, and then sort out how to host the JAR more reliably...

        I'll go revert the changes.

        Show
        Michael McCandless added a comment - OK I think we should simply revert our changes to the build.xml's, now, and then sort out how to host the JAR more reliably... I'll go revert the changes.
        Hide
        Michael McCandless added a comment -

        Moving to 3.0... this need not block 2.9.

        Show
        Michael McCandless added a comment - Moving to 3.0... this need not block 2.9.
        Hide
        Simon Willnauer added a comment -

        Moving out for 3.1

        Show
        Simon Willnauer added a comment - Moving out for 3.1
        Hide
        Simon Willnauer added a comment -

        I haven't looked at this issue for a while now but I figured today that the version we are using is "not available for download" anymore on the oracle pages. If you follow the link in the build file you will be able to download the zip file but I guess we should upgrade to the latest 3.3 version of BDB-JE.
        (see http://www.oracle.com/technology/software/products/berkeley-db/je/index.html - version 3.3.69)
        There is also another mirror that serves the jar directly (a maven repository) that might be more reliable.
        I updated the patch to load the 3.3.93 version of the jar directly and skip the unzip step as we now download only the jar file. I also updated the maven pom template files to reference the right version of BDB-JE which wasn't the case though.

        I think we should give the maven-repo mirror a chance though.

        Show
        Simon Willnauer added a comment - I haven't looked at this issue for a while now but I figured today that the version we are using is "not available for download" anymore on the oracle pages. If you follow the link in the build file you will be able to download the zip file but I guess we should upgrade to the latest 3.3 version of BDB-JE. (see http://www.oracle.com/technology/software/products/berkeley-db/je/index.html - version 3.3.69) There is also another mirror that serves the jar directly (a maven repository) that might be more reliable. I updated the patch to load the 3.3.93 version of the jar directly and skip the unzip step as we now download only the jar file. I also updated the maven pom template files to reference the right version of BDB-JE which wasn't the case though. I think we should give the maven-repo mirror a chance though.
        Hide
        Michael McCandless added a comment -

        Looks good Simon! Getting the JAR directly is a good savings (it's roughly 1 MB, but the full .zip distribution is roughly 6 MB. I agree we should give maven a shot... hopefully it's more reliable than the direct download we do now.

        Show
        Michael McCandless added a comment - Looks good Simon! Getting the JAR directly is a good savings (it's roughly 1 MB, but the full .zip distribution is roughly 6 MB. I agree we should give maven a shot... hopefully it's more reliable than the direct download we do now.
        Hide
        Simon Willnauer added a comment -

        mike, can you take this issue it unfortunately touches core stuff :/

        simon

        Show
        Simon Willnauer added a comment - mike, can you take this issue it unfortunately touches core stuff :/ simon
        Hide
        Michael McCandless added a comment -

        OK...

        Show
        Michael McCandless added a comment - OK...
        Hide
        Michael McCandless added a comment -

        Resolving this... if we still have reliability problems download the JAR from Oracle, let's open a new issue to find a better way to host the jar.

        Show
        Michael McCandless added a comment - Resolving this... if we still have reliability problems download the JAR from Oracle, let's open a new issue to find a better way to host the jar.
        Hide
        Simon Willnauer added a comment -

        mike, thanks for resolving this. I already replied to the commit mail but mention it here again for completeness....
        We should add a changes.txt entry to notify users that we upgraded the version.

        simon

        Show
        Simon Willnauer added a comment - mike, thanks for resolving this. I already replied to the commit mail but mention it here again for completeness.... We should add a changes.txt entry to notify users that we upgraded the version. simon
        Hide
        Michael McCandless added a comment -

        We should add a changes.txt entry to notify users that we upgraded the version.

        Ahh right – will do.

        Show
        Michael McCandless added a comment - We should add a changes.txt entry to notify users that we upgraded the version. Ahh right – will do.

          People

          • Assignee:
            Michael McCandless
            Reporter:
            Michael McCandless
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development