Hadoop Common
  1. Hadoop Common
  2. HADOOP-7412 Mavenization Umbrella
  3. HADOOP-7506

hadoopcommon build version cant be set from the maven commandline

    Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Not a Problem
    • Affects Version/s: 0.23.0
    • Fix Version/s: None
    • Component/s: build
    • Labels:
      None

      Description

      pom.xml had to introduce hadoop.version property with the default value set to the snapshot version. If someone during build time want to override the version from maven command line they can do so by passing -Dhadoop.version="". For ppl who doesnt want to change the default version can continue building.

      1. HADOOP-7506.PATCH
        3 kB
        Giridharan Kesavan

        Activity

        Hide
        Giridharan Kesavan added a comment -

        with this patch build version can be overridden from the commandline by using -Dhadoop.version=<version#>

        the default version is set to 0.203.0-SNAPSHOT

        Show
        Giridharan Kesavan added a comment - with this patch build version can be overridden from the commandline by using -Dhadoop.version=<version#> the default version is set to 0.203.0-SNAPSHOT
        Hide
        Alejandro Abdelnur added a comment -

        Note Maven warning on this:

        [WARNING] Some problems were encountered while building the effective model for org.apache.hadoop:hadoop-project:pom:0.23.0-SNAPSHOST
        [WARNING] 'version' contains an expression but should be a constant. @ org.apache.hadoop:hadoop-project:${hadoop.version}, /Users/tucu/src/apache/hadoop/utrunk/hadoop-project/pom.xml, line 19, column 12
        

        In addition, this won't work (I've tried this approach a couple of years ago when starting with Oozie) for the following reason:

        The installed/deployed POMs will have a variable as the version and Maven dependency resolution won't be able to resolve that variable to a concrete value. Following there is a minimal POM that fails after doing a 'mvn install' with the HADOOP-7506 patch. (Make sure you trash other snapshots of hadoop-* and you are using -offline, to force the use of the just installed artifacts).

        <?xml version="1.0" encoding="UTF-8"?>
        <project>
          <modelVersion>4.0.0</modelVersion>
          <groupId>x</groupId>
          <artifactId>x</artifactId>
          <version>1</version>
          <packaging>jar</packaging>
        
          <dependencies>
            <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>0.23.0-SNAPSHOT</version>
            </dependency>
          </dependencies>
        
        </project>
        

        One of the reasons for the failure is that the version is defined in hadoop-common's parent POM (hadoop-project) and then Maven cannot determine the version of hadoop-project to use because (chicken & egg problem).

        The Maven way to take care of version upgrades through out the POMs is using the maven-release-plugin (I've used it in the past and it works quite well in multi-module projects)

        Show
        Alejandro Abdelnur added a comment - Note Maven warning on this: [WARNING] Some problems were encountered while building the effective model for org.apache.hadoop:hadoop-project:pom:0.23.0-SNAPSHOST [WARNING] 'version' contains an expression but should be a constant. @ org.apache.hadoop:hadoop-project:${hadoop.version}, /Users/tucu/src/apache/hadoop/utrunk/hadoop-project/pom.xml, line 19, column 12 In addition, this won't work (I've tried this approach a couple of years ago when starting with Oozie) for the following reason: The installed/deployed POMs will have a variable as the version and Maven dependency resolution won't be able to resolve that variable to a concrete value. Following there is a minimal POM that fails after doing a 'mvn install' with the HADOOP-7506 patch. (Make sure you trash other snapshots of hadoop-* and you are using -offline, to force the use of the just installed artifacts). <?xml version= "1.0" encoding= "UTF-8" ?> <project> <modelVersion>4.0.0</modelVersion> <groupId>x</groupId> <artifactId>x</artifactId> <version>1</version> <packaging>jar</packaging> <dependencies> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-common</artifactId> <version>0.23.0-SNAPSHOT</version> </dependency> </dependencies> </project> One of the reasons for the failure is that the version is defined in hadoop-common's parent POM (hadoop-project) and then Maven cannot determine the version of hadoop-project to use because (chicken & egg problem). The Maven way to take care of version upgrades through out the POMs is using the maven-release-plugin (I've used it in the past and it works quite well in multi-module projects)
        Hide
        Luke Lu added a comment -

        The patch itself won't work for cross-module projects. Especially if downstream projects use ivy. We have to do some controlled version variable expansion like what I did for MR-279.

        However the maven-release-plugin way doesn't really work either, especially for CI, as I argued in MNG-624 and MNG-2971. Creating new tag/branch for each CI build (requires a different version instead of just something-SNAPSHOT) is extremely slow and unwieldy.

        Show
        Luke Lu added a comment - The patch itself won't work for cross-module projects. Especially if downstream projects use ivy. We have to do some controlled version variable expansion like what I did for MR-279. However the maven-release-plugin way doesn't really work either, especially for CI, as I argued in MNG-624 and MNG-2971 . Creating new tag/branch for each CI build (requires a different version instead of just something-SNAPSHOT) is extremely slow and unwieldy.
        Hide
        Alejandro Abdelnur added a comment -

        @Luke, I was not suggesting to use the maven-release-plugin for CI. I didn't know CI was the main motivation of the patch. Still for CI.

        The release plugin is quite handy when it comes to do a release of a multi-module project, taking care of tagging, updating to the release version all artifacts, deploying them, and updating the version to the new devel version.

        Not yet, but eventually we should use. Because of this, versions should not be parameterized, else the release plugin will not do its magic there.

        Maybe the versions plugin could be use for this purposes during the CI: http://mojo.codehaus.org/versions-maven-plugin/index.html

        Show
        Alejandro Abdelnur added a comment - @Luke, I was not suggesting to use the maven-release-plugin for CI. I didn't know CI was the main motivation of the patch. Still for CI. The release plugin is quite handy when it comes to do a release of a multi-module project, taking care of tagging, updating to the release version all artifacts, deploying them, and updating the version to the new devel version. Not yet, but eventually we should use. Because of this, versions should not be parameterized, else the release plugin will not do its magic there. Maybe the versions plugin could be use for this purposes during the CI: http://mojo.codehaus.org/versions-maven-plugin/index.html
        Hide
        Allen Wittenauer added a comment -

        If we can't change the version # at build time, I don't think we'll be able to upgrade server side-only components without also upgrading all the clients. That's a major hit on the ops side. If that holds true, then we'll need to back out the maven patch before release if we can't fix this.

        Show
        Allen Wittenauer added a comment - If we can't change the version # at build time, I don't think we'll be able to upgrade server side-only components without also upgrading all the clients. That's a major hit on the ops side. If that holds true, then we'll need to back out the maven patch before release if we can't fix this.
        Hide
        Luke Lu added a comment -

        Maybe the versions plugin could be use for this purposes during the CI: http://mojo.codehaus.org/versions-maven-plugin/index.html

        Just verified that mvn versions:set -DnewVersion=0.23.0-my-version works without the scm stuff. So this is a viable way for CI build, though still less convenient than version properties which can be easily controlled by profiles (e.g., via settings.xml)

        Show
        Luke Lu added a comment - Maybe the versions plugin could be use for this purposes during the CI: http://mojo.codehaus.org/versions-maven-plugin/index.html Just verified that mvn versions:set -DnewVersion=0.23.0-my-version works without the scm stuff. So this is a viable way for CI build, though still less convenient than version properties which can be easily controlled by profiles (e.g., via settings.xml)
        Hide
        Scott Carey added a comment -

        I have given up on the maven-release-plugin. It fails in many situations, too many of them to list here. If you want to do something like have trunk named trunk-SNAPSHOT and branches named in custom ways, good luck. It is an example of a design anti-pattern: it is a conflated design. It tries to do too much at once.

        Do one thing and do it well:

        Use http://mojo.codehaus.org/versions-maven-plugin/ to update the version of items in multi-module products.
        Use plain old svn command line to branch/tag.
        Use plain old maven to verify a build.

        Show
        Scott Carey added a comment - I have given up on the maven-release-plugin. It fails in many situations, too many of them to list here. If you want to do something like have trunk named trunk-SNAPSHOT and branches named in custom ways, good luck. It is an example of a design anti-pattern: it is a conflated design. It tries to do too much at once. Do one thing and do it well: Use http://mojo.codehaus.org/versions-maven-plugin/ to update the version of items in multi-module products. Use plain old svn command line to branch/tag. Use plain old maven to verify a build.
        Hide
        Giridharan Kesavan added a comment -

        Luke's suggestion seem to work. Thanks Luke.

        mvn versions:set -DnewVersion=<newversion}
        need to execute the same command at two levels
        executing at trunk level changes the version in the root pom.xml and hadoop-common/pom.xml
        executing the command at trunk/hadoop-project level updates the version for hadoop-project, hadoop-annotations and hadoop-assemblies pom.xml
        
        Show
        Giridharan Kesavan added a comment - Luke's suggestion seem to work. Thanks Luke. mvn versions:set -DnewVersion=<newversion} need to execute the same command at two levels executing at trunk level changes the version in the root pom.xml and hadoop-common/pom.xml executing the command at trunk/hadoop-project level updates the version for hadoop-project, hadoop-annotations and hadoop-assemblies pom.xml
        Hide
        Tom White added a comment -

        Giri - can this be closed now you've got a solution?

        Show
        Tom White added a comment - Giri - can this be closed now you've got a solution?
        Hide
        Tom White added a comment -

        Closing since there's a workaround.

        Show
        Tom White added a comment - Closing since there's a workaround.

          People

          • Assignee:
            Giridharan Kesavan
            Reporter:
            Giridharan Kesavan
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development