Hadoop Common
  1. Hadoop Common
  2. HADOOP-6382

publish hadoop jars to apache mvn repo.

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.0
    • Fix Version/s: 0.20.3
    • Component/s: build
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      The hadoop jars are renamed from previous hadoop-<version>-<name>.jar to hadoop-<name>-<version>.jar. Applications and documentation need to be updated to use the new file naming scheme.

      Description

      use maven ant task to publish hadoop 20 jars to the apache maven repo

      1. h20-v3tov5.patch
        4 kB
        Lee Tucker
      2. hadoop-6382.patch
        28 kB
        Giridharan Kesavan
      3. HADOOP-6382.script.patch
        1.0 kB
        Lee Tucker
      4. HADOOP-6382-findbugs.patch
        1 kB
        Lee Tucker
      5. HADOOP-6382-findbugs.patch
        0.9 kB
        Lee Tucker
      6. hadoop-6382-v1.patch
        40 kB
        Giridharan Kesavan
      7. hadoop-6382-v2.patch
        41 kB
        Lee Tucker
      8. hadoop-6382-v3.patch
        42 kB
        Lee Tucker
      9. hadoop-6382-v4.patch
        50 kB
        Giridharan Kesavan
      10. hadoop-6382-v5.patch
        50 kB
        Giridharan Kesavan

        Issue Links

          Activity

          Hide
          Giridharan Kesavan added a comment -

          this patch has the mvn-ant-task to publish hadoop jars to the apache mvn repo.

          P.S: patch has some basic poms, I would be workin with the dev team get the right set of pom's

          Show
          Giridharan Kesavan added a comment - this patch has the mvn-ant-task to publish hadoop jars to the apache mvn repo. P.S: patch has some basic poms, I would be workin with the dev team get the right set of pom's
          Hide
          Giridharan Kesavan added a comment -

          updated the affected versions field:

          Show
          Giridharan Kesavan added a comment - updated the affected versions field:
          Hide
          Lee Tucker added a comment -

          Given that the new xml files are actually poms (for maven in ant) shouldn't that be reflected in the file names, hadoop-core-pom.xml instead of hadoop-core-template.xml. I know it's a nit, but it will make it clearer that these files are POMs

          Show
          Lee Tucker added a comment - Given that the new xml files are actually poms (for maven in ant) shouldn't that be reflected in the file names, hadoop-core-pom.xml instead of hadoop-core-template.xml. I know it's a nit, but it will make it clearer that these files are POMs
          Hide
          Chad Metcalf added a comment -

          Whats the status on this? I'm in to help where required.

          Show
          Chad Metcalf added a comment - Whats the status on this? I'm in to help where required.
          Hide
          Giridharan Kesavan added a comment -

          ant mvn-install

          • To install artifact to the local filesystem

          ant mvn-deploy

          • To deploy snapshots to the apache nexus repo (looks for authentication in the ~/.m2/settings.xml)

          ant mvn-deploy -Drepo=staging -Dversion=<maven standard version number>

          • To deploy artifacts for voting before release , this also requires authentication configured in ~/.m2/settings.xml

          Deploying artifacts to the staging repository requires signing the artifacts with gpg keys, mvn-deploy target takes care of
          signing the artifacts. While executing mvn-deploy target with -Drepo=staging it would ask for gpg passphrase
          which need to be keyed in.

          Once the deployment is successful, to make the artifact available in the staging repository , login into the staging repository and close the staging by right clicking on the staged artifact at http:/repository.apache.org

          With this patch i ve already uploaded artifacts to the stating repository; (only ppl with committer access would be able to view this, as the repository is not closed yet)

          Thanks

          Show
          Giridharan Kesavan added a comment - ant mvn-install To install artifact to the local filesystem ant mvn-deploy To deploy snapshots to the apache nexus repo (looks for authentication in the ~/.m2/settings.xml) ant mvn-deploy -Drepo=staging -Dversion=<maven standard version number> To deploy artifacts for voting before release , this also requires authentication configured in ~/.m2/settings.xml Deploying artifacts to the staging repository requires signing the artifacts with gpg keys, mvn-deploy target takes care of signing the artifacts. While executing mvn-deploy target with -Drepo=staging it would ask for gpg passphrase which need to be keyed in. Once the deployment is successful, to make the artifact available in the staging repository , login into the staging repository and close the staging by right clicking on the staged artifact at http:/repository.apache.org With this patch i ve already uploaded artifacts to the stating repository; (only ppl with committer access would be able to view this, as the repository is not closed yet) Thanks
          Hide
          steve_l added a comment -

          do you want any of the committers to do something here? Are there any artifacts + metadata that we are happy to formally release?

          Show
          steve_l added a comment - do you want any of the committers to do something here? Are there any artifacts + metadata that we are happy to formally release?
          Hide
          Giridharan Kesavan added a comment -

          I'm not sure of any formal release at this time.
          But, if we can review and get this committed we would be release ready(from the artifacts publishing front)

          Show
          Giridharan Kesavan added a comment - I'm not sure of any formal release at this time. But, if we can review and get this committed we would be release ready(from the artifacts publishing front)
          Hide
          steve_l added a comment -

          I'd go for testing this the next beta release we do of anything, just to test the full end-to-end release process. You aren't allowed to change the metadata on published artifacts (most clients cache them and never poll for changes), so releasing JARS for beta-versions helps debug the release process.

          Show
          steve_l added a comment - I'd go for testing this the next beta release we do of anything, just to test the full end-to-end release process. You aren't allowed to change the metadata on published artifacts (most clients cache them and never poll for changes), so releasing JARS for beta-versions helps debug the release process.
          Hide
          Lee Tucker added a comment -

          This fix is necessary to correct an error in hadoop-6382-v1.patch which has been previously supplied to the yahoo-hadoop-0.20 branch

          Show
          Lee Tucker added a comment - This fix is necessary to correct an error in hadoop-6382-v1.patch which has been previously supplied to the yahoo-hadoop-0.20 branch
          Hide
          Lee Tucker added a comment -

          this patch fixes a fault in the previous patch for findbugs which was not detected until after the commit.

          Show
          Lee Tucker added a comment - this patch fixes a fault in the previous patch for findbugs which was not detected until after the commit.
          Hide
          Lee Tucker added a comment -

          Further inspection of the patch revealed at least 2 other locations where wildcarding of jar names followed the non-maven convention. I've corrected those locations as well for yahoo-hadoop-0.20. Now I'll build a clean patch for external 20.

          Show
          Lee Tucker added a comment - Further inspection of the patch revealed at least 2 other locations where wildcarding of jar names followed the non-maven convention. I've corrected those locations as well for yahoo-hadoop-0.20. Now I'll build a clean patch for external 20.
          Hide
          Lee Tucker added a comment -

          correct the branch-0.20 version of the patch to correct for all locations where jar file naming convention changed.

          Show
          Lee Tucker added a comment - correct the branch-0.20 version of the patch to correct for all locations where jar file naming convention changed.
          Hide
          Lee Tucker added a comment -

          Updating the jars caused the bin scripts to break, this was only found after deployment of the built objects. A further uberpatch will be uploaded shortly. But this patch will patch the previous large patch to make the bin scripts function. If you have a local hadoop-env.sh script which adds a scheduler jar, you'll need to adjust it accordingly.

          Show
          Lee Tucker added a comment - Updating the jars caused the bin scripts to break, this was only found after deployment of the built objects. A further uberpatch will be uploaded shortly. But this patch will patch the previous large patch to make the bin scripts function. If you have a local hadoop-env.sh script which adds a scheduler jar, you'll need to adjust it accordingly.
          Hide
          Lee Tucker added a comment -

          Uberpatch with both -v2 and the necessary changes to the bin files.

          Show
          Lee Tucker added a comment - Uberpatch with both -v2 and the necessary changes to the bin files.
          Hide
          Giridharan Kesavan added a comment -

          This v4 patch adds hadoop-core dependency in the tools and examples pom file.

          Show
          Giridharan Kesavan added a comment - This v4 patch adds hadoop-core dependency in the tools and examples pom file.
          Hide
          Benson Margulies added a comment -

          At Chris Douglas' request I gave this a read. The ant-running-maven stuff, which I've used before, looks reasonable to me. I haven't done this with nexus pro and staging, so my review doesn't help you validate that.

          Show
          Benson Margulies added a comment - At Chris Douglas' request I gave this a read. The ant-running-maven stuff, which I've used before, looks reasonable to me. I haven't done this with nexus pro and staging, so my review doesn't help you validate that.
          Hide
          Benson Margulies added a comment -

          At Chris Douglas' request I gave this a read. The ant-running-maven stuff, which I've used before, looks reasonable to me. I haven't done this with nexus pro and staging, so my review doesn't help you validate that.

          Show
          Benson Margulies added a comment - At Chris Douglas' request I gave this a read. The ant-running-maven stuff, which I've used before, looks reasonable to me. I haven't done this with nexus pro and staging, so my review doesn't help you validate that.
          Hide
          Benson Margulies added a comment -

          At Chris Douglas' request I gave this a read. The ant-running-maven stuff, which I've used before, looks reasonable to me. I haven't done this with nexus pro and staging, so my review doesn't help you validate that.

          Show
          Benson Margulies added a comment - At Chris Douglas' request I gave this a read. The ant-running-maven stuff, which I've used before, looks reasonable to me. I haven't done this with nexus pro and staging, so my review doesn't help you validate that.
          Hide
          ryan rawson added a comment -

          this looks a lot better than what is in 0.20.2, no broken dependencies, etc. The latest patch doesnt generate -javadoc.jar or -sources.jar though. That is very handy for people who are in IDEs.

          Show
          ryan rawson added a comment - this looks a lot better than what is in 0.20.2, no broken dependencies, etc. The latest patch doesnt generate -javadoc.jar or -sources.jar though. That is very handy for people who are in IDEs.
          Hide
          Giridharan Kesavan added a comment -

          If everyone is okay with this approach, we could commit this patch and file a jira for getting -javadoc.jar and -sources.jar

          Show
          Giridharan Kesavan added a comment - If everyone is okay with this approach, we could commit this patch and file a jira for getting -javadoc.jar and -sources.jar
          Hide
          Chris Douglas added a comment -

          Very minor, largely cosmetic nits:

          • This should be removed:
            +    <echo> phassphrase : ${gpg.passphrase} </echo>
            
          • The pom template properties have odd names:
            +  <property name="hadoop-core-pom-template.xml" 
            +    location="${ivy.dir}/hadoop-core-pom-template.xml"/>
            

            ({{$

            {filename.xml}

            }} looks like an error). Leaving off the .xml suffix and following the convention of . rather than - as a separator in property names would help readers parse these sections.

          • Existing templates are named foobar.xml.template rather than foobar-template.xml. Is there a reason for the different naming scheme?
          • The hadoop.jar property is no longer used and can be removed or should use the new name for the core jar:
            -  <property name="hadoop.jar" location="${build.dir}/hadoop-${hadoop.version}-core.jar" />
            +  <property name="hadoop.jar" 
            +    location="${build.dir}/hadoop-${hadoop.version}-core.jar"/>
            
          • Deleting all .asc files in the tree rather than using a common list for signing/deleting may be too aggressive. However, relaxing this can be an enhancement and doesn't block commit

          According to Lee, that the pom templates are static- and don't update versions with ivy, used for versioning dependencies elsewhere- is by design for this pass.

          I agree that adding the javadoc and sources jars can be pushed to a separate issue (since this will also need to happen on trunk). The changes complying with maven naming conventions require that this be marked as an incompatible change (the build now generates hadoop-<name>-<version>.jar vs previous (hadoop-<version>-<name>.jar), so this needs a release note calling that out.

          Other than these, the patch looks good and its effect in staging has been confirmed by members of the Mahout team (thanks Benson and Sean), and it worked for the 0.20.2 release. Thanks, Giri; this is excellent.

          Show
          Chris Douglas added a comment - Very minor, largely cosmetic nits: This should be removed: + <echo> phassphrase : ${gpg.passphrase} </echo> The pom template properties have odd names: + <property name="hadoop-core-pom-template.xml" + location="${ivy.dir}/hadoop-core-pom-template.xml"/> ({{$ {filename.xml} }} looks like an error). Leaving off the .xml suffix and following the convention of . rather than - as a separator in property names would help readers parse these sections. Existing templates are named foobar.xml.template rather than foobar-template.xml . Is there a reason for the different naming scheme? The hadoop.jar property is no longer used and can be removed or should use the new name for the core jar: - <property name="hadoop.jar" location="${build.dir}/hadoop-${hadoop.version}-core.jar" /> + <property name="hadoop.jar" + location="${build.dir}/hadoop-${hadoop.version}-core.jar"/> Deleting all .asc files in the tree rather than using a common list for signing/deleting may be too aggressive. However, relaxing this can be an enhancement and doesn't block commit According to Lee, that the pom templates are static- and don't update versions with ivy, used for versioning dependencies elsewhere- is by design for this pass. I agree that adding the javadoc and sources jars can be pushed to a separate issue (since this will also need to happen on trunk). The changes complying with maven naming conventions require that this be marked as an incompatible change (the build now generates hadoop-<name>-<version>.jar vs previous ( hadoop-<version>-<name>.jar ), so this needs a release note calling that out. Other than these, the patch looks good and its effect in staging has been confirmed by members of the Mahout team (thanks Benson and Sean), and it worked for the 0.20.2 release. Thanks, Giri; this is excellent.
          Hide
          Giridharan Kesavan added a comment -

          Fixes most of the comments mentioned, Thanks Chris.

          Show
          Giridharan Kesavan added a comment - Fixes most of the comments mentioned, Thanks Chris.
          Hide
          Chris Douglas added a comment -

          +1

          I committed this. Thanks, Giri!

          Show
          Chris Douglas added a comment - +1 I committed this. Thanks, Giri!
          Hide
          Patrick Angeles added a comment -

          Patch here to include source jars:

          https://issues.apache.org/jira/browse/HADOOP-6635

          It's a much more invasive procedure to add javadoc jars since the build aggregates javadocs from all sub-projects into a single tree. Besides, you get 'javadocs' included in the source comments so the benefits are marginal

          Show
          Patrick Angeles added a comment - Patch here to include source jars: https://issues.apache.org/jira/browse/HADOOP-6635 It's a much more invasive procedure to add javadoc jars since the build aggregates javadocs from all sub-projects into a single tree. Besides, you get 'javadocs' included in the source comments so the benefits are marginal
          Hide
          Lee Tucker added a comment -

          yahoo-hadoop-0.20 patch for early commit of this issue.

          Show
          Lee Tucker added a comment - yahoo-hadoop-0.20 patch for early commit of this issue.
          Hide
          Patrick Angeles added a comment -

          I'm curious if anyone's managed to deploy to the apache repo using HTTP.

          It says here that the wagon-http provider doesn't support deployment:

          http://maven.apache.org/wagon/wagon-providers/wagon-http/

          Show
          Patrick Angeles added a comment - I'm curious if anyone's managed to deploy to the apache repo using HTTP. It says here that the wagon-http provider doesn't support deployment: http://maven.apache.org/wagon/wagon-providers/wagon-http/
          Hide
          Giridharan Kesavan added a comment -

          wagon plugin supports both http and https but we can only publish to the apache nexus repo using https.

          Show
          Giridharan Kesavan added a comment - wagon plugin supports both http and https but we can only publish to the apache nexus repo using https.
          Hide
          Eli Collins added a comment -
          • Marking as an incompatible change since this patch changed how the hadoop jars are named.
          • Adding a release note.
          • I filed HADOOP-6844 to update the docs to reflect this change.
          Show
          Eli Collins added a comment - Marking as an incompatible change since this patch changed how the hadoop jars are named. Adding a release note. I filed HADOOP-6844 to update the docs to reflect this change.

            People

            • Assignee:
              Giridharan Kesavan
              Reporter:
              Giridharan Kesavan
            • Votes:
              6 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development