Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: general/build
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      There are a couple of things we can improve for the next release:

      • "*pom.xml" files should be renamed to "*pom.xml.template"
      • artifacts "lucene-parent" should extend "apache-parent"
      • add source jars as artifacts
      • update <generate-maven-artifacts> task to work with latest version of maven-ant-tasks.jar
      • metadata filenames should not contain "local"
      1. lucene-935.patch
        21 kB
        Michael Busch
      2. lucene-935-new.patch
        1 kB
        Michael Busch
      3. lucene-935-remote-repos.patch
        1 kB
        Michael Busch
      4. lucene-935-rename-poms.patch
        84 kB
        Michael Busch

        Issue Links

          Activity

          Hide
          michaelbusch Michael Busch added a comment -

          This patch:

          • Includes the latest patch from LUCENE-908
          • Uses the maven ant task <artifact:deploy> instead of <artifact:install>.
            The advantage is that the metadata files don't contain "local" in the
            filenames anymore. This also solves some problems with the latest version
            of the maven ant tasks and simplifies the <generate-maven-artifacts> target
            in contrib-build.xml.
          • The macro <m2-deploy> now has a nested, optional element
            <artifact-attachments> which is used to attach the sources as jar files if
            possible for an artifact.
          • The macro <jarify> now has two attributes "destfile" and "basedir" which
            allows <jarify> to be used by other targets like <jar-src>.
          • The target <jar-src> now calls the macro <jarify> which makes it possible to
            overwrite <jar-src> in contrib build.xml files. This is neccessary to include
            special license files in the meta-inf folder (needed for snowball) or to add
            additional content to the MANIFEST file (needed for lucli).
          • "lucene-parent" now extends artifact "apache"

          Note: I separated the patch into two files to improve readability. Both patches
          have to be applied.

          Show
          michaelbusch Michael Busch added a comment - This patch: Includes the latest patch from LUCENE-908 Uses the maven ant task <artifact:deploy> instead of <artifact:install>. The advantage is that the metadata files don't contain "local" in the filenames anymore. This also solves some problems with the latest version of the maven ant tasks and simplifies the <generate-maven-artifacts> target in contrib-build.xml. The macro <m2-deploy> now has a nested, optional element <artifact-attachments> which is used to attach the sources as jar files if possible for an artifact. The macro <jarify> now has two attributes "destfile" and "basedir" which allows <jarify> to be used by other targets like <jar-src>. The target <jar-src> now calls the macro <jarify> which makes it possible to overwrite <jar-src> in contrib build.xml files. This is neccessary to include special license files in the meta-inf folder (needed for snowball) or to add additional content to the MANIFEST file (needed for lucli). "lucene-parent" now extends artifact "apache" Note: I separated the patch into two files to improve readability. Both patches have to be applied.
          Hide
          michaelbusch Michael Busch added a comment -

          I'm planning to commit this together with LUCENE-908 in a day or so...

          Show
          michaelbusch Michael Busch added a comment - I'm planning to commit this together with LUCENE-908 in a day or so...
          Hide
          michaelbusch Michael Busch added a comment -

          Committed. Revision: 568766

          Show
          michaelbusch Michael Busch added a comment - Committed. Revision: 568766
          Hide
          michaelbusch Michael Busch added a comment -
          • Define property "m2.repository.url" that defaults to "file://$ {maven.dist.dir}

            " and use it
            in m2-deploy.

          • Register provider needed for using the transfer protocol of the remote repository.
            Example for scp: <artifact:install-provider artifactId="wagon-ssh" version="1.0-beta-2"/>
          Show
          michaelbusch Michael Busch added a comment - Define property "m2.repository.url" that defaults to "file://$ {maven.dist.dir} " and use it in m2-deploy. Register provider needed for using the transfer protocol of the remote repository. Example for scp: <artifact:install-provider artifactId="wagon-ssh" version="1.0-beta-2"/>
          Hide
          michaelbusch Michael Busch added a comment -

          Does our build machine have local access to /www/people.apache.org/maven-snapshot-repository ? I'm asking this question because I'm not familiar with hudson and the build infrastructure. If the answer is yes, then we don't have to register any special providers to deploy to the snapshot repository.

          However, if the answer is no, how can we deploy to the repository? Remotely via scp?

          Show
          michaelbusch Michael Busch added a comment - Does our build machine have local access to /www/people.apache.org/maven-snapshot-repository ? I'm asking this question because I'm not familiar with hudson and the build infrastructure. If the answer is yes, then we don't have to register any special providers to deploy to the snapshot repository. However, if the answer is no, how can we deploy to the repository? Remotely via scp?
          Hide
          buschmic Michael Busch added a comment -

          This small patch adds the property "m2.repository.url", that defaults to
          "file://$

          {maven.dist.dir}

          ". You easily can set a different value:
          ant -Dm2.repository.url="file://C:\temp\maven"

          I'm going to commit this now. In case our build machine cannot deploy to
          the snapshot repository using the local path
          "file:///www/people.apache.org/maven-snapshot-repository" let me know
          please, then we'll have to register a provider for an appropriate transfer
          protocol.

          Show
          buschmic Michael Busch added a comment - This small patch adds the property "m2.repository.url", that defaults to "file://$ {maven.dist.dir} ". You easily can set a different value: ant -Dm2.repository.url="file://C:\temp\maven" I'm going to commit this now. In case our build machine cannot deploy to the snapshot repository using the local path "file:///www/people.apache.org/maven-snapshot-repository" let me know please, then we'll have to register a provider for an appropriate transfer protocol.
          Hide
          michaelbusch Michael Busch added a comment -

          I just committed the latest patch.

          I'm leaving this open in case more work needs to be done in order to deploy to the snapshot repository.
          I'm clearing the fix version though.

          Show
          michaelbusch Michael Busch added a comment - I just committed the latest patch. I'm leaving this open in case more work needs to be done in order to deploy to the snapshot repository. I'm clearing the fix version though.
          Hide
          gsingers Grant Ingersoll added a comment -

          I have the necessary pieces in place for this, just need to figure out some security issues between zones and p.a.o.

          For now, you can see a sample of what should be published at http://people.apache.org/maven-snapshot-repository/org/apache/lucene/ based on me running it by hand.

          I will update when the security issues are straightened out. So just know that those jars there are based on trunk from this morning (11/1/07) until further notice.

          Show
          gsingers Grant Ingersoll added a comment - I have the necessary pieces in place for this, just need to figure out some security issues between zones and p.a.o. For now, you can see a sample of what should be published at http://people.apache.org/maven-snapshot-repository/org/apache/lucene/ based on me running it by hand. I will update when the security issues are straightened out. So just know that those jars there are based on trunk from this morning (11/1/07) until further notice.
          Hide
          michaelbusch Michael Busch added a comment -

          Grant,
          two comments:

          • How often are you planning two publish a snapshot? In every nightly build? Or in every nightly build unless no commit happened? Seems too often, I took a look a the other projects that publish snapshots in that repository and they don't seem to publish more often than once per month.
          • For publishing a snapshot we should set the version to "x.y.z-SNAPSHOT" and increase z for every snapshot?
          Show
          michaelbusch Michael Busch added a comment - Grant, two comments: How often are you planning two publish a snapshot? In every nightly build? Or in every nightly build unless no commit happened? Seems too often, I took a look a the other projects that publish snapshots in that repository and they don't seem to publish more often than once per month. For publishing a snapshot we should set the version to "x.y.z-SNAPSHOT" and increase z for every snapshot?
          Hide
          gsingers Grant Ingersoll added a comment -

          well, I don't know that I have a lot of control over it unless I want to spend a good deal of time scripting it, etc. (which I don't have the time to do)

          I guess, ideally, we would only publish on commit, but I am not sure how to enable that at the moment. I wish we could just run the Maven deploy target for the nightly build and be done with it (instead of all these ANT gyrations)

          At any rate, right now it is setup to do nightly regardless and the VERSION ID is generated by Hudson, so it should mirror the version numbers for the nightly builds there.

          Let's let it run tonight as part of the nightly build and see how it goes. Then we can decide on frequency, etc.

          Show
          gsingers Grant Ingersoll added a comment - well, I don't know that I have a lot of control over it unless I want to spend a good deal of time scripting it, etc. (which I don't have the time to do) I guess, ideally, we would only publish on commit, but I am not sure how to enable that at the moment. I wish we could just run the Maven deploy target for the nightly build and be done with it (instead of all these ANT gyrations) At any rate, right now it is setup to do nightly regardless and the VERSION ID is generated by Hudson, so it should mirror the version numbers for the nightly builds there. Let's let it run tonight as part of the nightly build and see how it goes. Then we can decide on frequency, etc.
          Hide
          gsingers Grant Ingersoll added a comment -

          Looks like it went through successfully last night. I suppose we should look up if there are any policies on how often to publish snapshots.

          Show
          gsingers Grant Ingersoll added a comment - Looks like it went through successfully last night. I suppose we should look up if there are any policies on how often to publish snapshots.
          Hide
          gsingers Grant Ingersoll added a comment -

          I'm thinking maybe what we should do is basically what we do on zones: only keep the last 10 builds. I also will try to modify the version id so it only has the date in it and not the time.

          Show
          gsingers Grant Ingersoll added a comment - I'm thinking maybe what we should do is basically what we do on zones: only keep the last 10 builds. I also will try to modify the version id so it only has the date in it and not the time.
          Hide
          michaelbusch Michael Busch added a comment -

          I think in nightly.sh we should call "ant generate-maven-artifacts" before we run clover code coverage?

          Otherwise the snapshots probably contain the clover instrumentation?

          Show
          michaelbusch Michael Busch added a comment - I think in nightly.sh we should call "ant generate-maven-artifacts" before we run clover code coverage? Otherwise the snapshots probably contain the clover instrumentation?
          Hide
          gsingers Grant Ingersoll added a comment -

          Done.

          Show
          gsingers Grant Ingersoll added a comment - Done.
          Hide
          gsingers Grant Ingersoll added a comment -

          I have changed the nightly.sh to output these now as 2.3-SNAPSHOT. I don't know how to get it to recognize the version number in the build file with out somehow slurping it from that file (not worth the effort), so instead, I will add to the release notes that the nightly.sh file needs to be updated upon release

          Show
          gsingers Grant Ingersoll added a comment - I have changed the nightly.sh to output these now as 2.3-SNAPSHOT. I don't know how to get it to recognize the version number in the build file with out somehow slurping it from that file (not worth the effort), so instead, I will add to the release notes that the nightly.sh file needs to be updated upon release
          Hide
          gsingers Grant Ingersoll added a comment -

          Added the note to the release TODO on the Wiki

          Show
          gsingers Grant Ingersoll added a comment - Added the note to the release TODO on the Wiki
          Hide
          michaelbusch Michael Busch added a comment -

          Hey Grant,

          can you try out this patch. It should make our life easier,
          I think I got the remote repos deployment to work in the
          ant script.

          Try this (just replace my username) :
          ant -Dm2.repository.url=scp://people.apache.org:/www/people.apache.org/maven-snapshot-repository -Dm2.repository.username=buschmi -Dm2.repository.private.key=/export/home/buschmi/.ssh/id_dsa -Dversion=2.3-SNAPSHOT clean generate-maven-artifacts

          Show
          michaelbusch Michael Busch added a comment - Hey Grant, can you try out this patch. It should make our life easier, I think I got the remote repos deployment to work in the ant script. Try this (just replace my username) : ant -Dm2.repository.url=scp://people.apache.org:/www/people.apache.org/maven-snapshot-repository -Dm2.repository.username=buschmi -Dm2.repository.private.key=/export/home/buschmi/.ssh/id_dsa -Dversion=2.3-SNAPSHOT clean generate-maven-artifacts
          Hide
          michaelbusch Michael Busch added a comment -

          I just committed the latest patch.

          OK, I think I can close this issue now. We have made good progress
          with the maven artifacts since we released 2.2. The artifacts include
          the sources and javadocs now and we're deploying nightly snapshots
          to the m2 snapshot repository.

          Thanks, Grant, and everyone else who helped here!!

          Show
          michaelbusch Michael Busch added a comment - I just committed the latest patch. OK, I think I can close this issue now. We have made good progress with the maven artifacts since we released 2.2. The artifacts include the sources and javadocs now and we're deploying nightly snapshots to the m2 snapshot repository. Thanks, Grant, and everyone else who helped here!!
          Hide
          gsingers Grant Ingersoll added a comment - - edited

          can you try out this patch

          BUILD FAILED
          ..../lucene/java/lucene935/build.xml:459: Specify at least one source - a file or a resource collection.

          This happened after lucene-xml-query-parser. It seems like most everything went through up to that point and I don't see any other errors.

          Show
          gsingers Grant Ingersoll added a comment - - edited can you try out this patch BUILD FAILED ..../lucene/java/lucene935/build.xml:459: Specify at least one source - a file or a resource collection. This happened after lucene-xml-query-parser. It seems like most everything went through up to that point and I don't see any other errors.
          Hide
          michaelbusch Michael Busch added a comment -

          Oups, I forgot to remove the lines that generate the checksums for the
          artifacts. <artifact:deploy> does this automatically. I committed the fix.

          I tried it out on zones, the build succeeds now.

          Show
          michaelbusch Michael Busch added a comment - Oups, I forgot to remove the lines that generate the checksums for the artifacts. <artifact:deploy> does this automatically. I committed the fix. I tried it out on zones, the build succeeds now.
          Hide
          gsingers Grant Ingersoll added a comment - - edited

          OK, now we just need to figure out how best to incorporate it into nightly.sh. Part of the problem is Hudson is the account running nightly.sh and it doesn't have an account on p.a.o. Thus, the need for Hudson to access a private key of one of us w/ a zones account. I don't really like this idea.

          Show
          gsingers Grant Ingersoll added a comment - - edited OK, now we just need to figure out how best to incorporate it into nightly.sh. Part of the problem is Hudson is the account running nightly.sh and it doesn't have an account on p.a.o. Thus, the need for Hudson to access a private key of one of us w/ a zones account. I don't really like this idea.
          Hide
          michaelbusch Michael Busch added a comment -

          generate-maven-artifacts still works fine if no m2.* properties are overridden. It
          then deploys locally to <dist_dir>/maven as before.

          So if the account doesn't have access to p.a.o, how does it copy the artifacts then?
          Do we use different accounts to run nightly.sh and the cron job currently?

          Show
          michaelbusch Michael Busch added a comment - generate-maven-artifacts still works fine if no m2.* properties are overridden. It then deploys locally to <dist_dir>/maven as before. So if the account doesn't have access to p.a.o, how does it copy the artifacts then? Do we use different accounts to run nightly.sh and the cron job currently?
          Hide
          gsingers Grant Ingersoll added a comment -

          My cron job copies from the Hudson dir to p.a.o. Whereas the Hudson script runs under a different account

          I realize this isn't great, but we asked infrastructure for a headless acct for Hudson on p.a.o and it was denied.

          I think for now, we can just leave it as is.

          Show
          gsingers Grant Ingersoll added a comment - My cron job copies from the Hudson dir to p.a.o. Whereas the Hudson script runs under a different account I realize this isn't great, but we asked infrastructure for a headless acct for Hudson on p.a.o and it was denied. I think for now, we can just leave it as is.

            People

            • Assignee:
              michaelbusch Michael Busch
              Reporter:
              michaelbusch Michael Busch
            • Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development