Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2432

MR-279: Install sanitized poms for downstream sanity

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.0
    • Component/s: mrv2
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Tags:
      mr-279, mrv2, pom

      Description

      Due to MNG-4223, the installed POMs of MR-279 is downstream hostile. E.g., it's impossible to use versions of hadoop-mapreduce-client-core.version in ivy other than 1.0-SNAPSHOT without changing the multiple POMs, rendering the version properties (hadoop-mapreduce.version and yarn.version) practically useless.

      This patch will install POMs with version (only) properties expanded. This patch also use inheritance and dependencyManagement to make POMs DRYer. It could use further cleanup to reduce "unnecessary" dependencies in some modules, but it's a working start.

      To see the patch work, apply the patch and do a mvn clean install -P-cbuild -DskipTests to make sure sane POMs are installed and then working on individual test issues.

      1. mr-2432-poms-v1.patch
        60 kB
        Luke Lu
      2. mr-2432-poms-v2.patch
        60 kB
        Luke Lu
      3. mr-2432-poms-v3.patch
        60 kB
        Luke Lu

        Activity

        Hide
        Mahadev konar added a comment -

        luke,
        I am seeing the following error with this patch:

        Missing:
        ----------
        1) org.apache.hadoop:hadoop-hdfs:jar:0.22-SNAPSHOT
        
          Try downloading the file manually from the project website.
        
          Then, install it using the command: 
              mvn install:install-file -DgroupId=org.apache.hadoop -DartifactId=hadoop-hdfs -Dversion=0.22-SNAPSHOT -Dpackaging=jar -Dfile=/path/to/file
        
          Alternatively, if you host your own repository you can deploy the file there: 
              mvn deploy:deploy-file -DgroupId=org.apache.hadoop -DartifactId=hadoop-hdfs -Dversion=0.22-SNAPSHOT -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
        
          Path to dependency: 
                1) org.apache.hadoop:hadoop-mapreduce-client-core:jar:1.0-SNAPSHOT
                2) org.apache.hadoop:hadoop-hdfs:jar:0.22-SNAPSHOT
        
        ----------
        1 required artifact is missing.
        
        for artifact: 
          org.apache.hadoop:hadoop-mapreduce-client-core:jar:1.0-SNAPSHOT
        
        from the specified remote repositories:
          central (http://repo1.maven.org/maven2),
          repository.jboss.org (http://repository.jboss.org/nexus/content/groups/public/)
        
        Show
        Mahadev konar added a comment - luke, I am seeing the following error with this patch: Missing: ---------- 1) org.apache.hadoop:hadoop-hdfs:jar:0.22-SNAPSHOT Try downloading the file manually from the project website. Then, install it using the command: mvn install:install-file -DgroupId=org.apache.hadoop -DartifactId=hadoop-hdfs -Dversion=0.22-SNAPSHOT -Dpackaging=jar -Dfile=/path/to/file Alternatively, if you host your own repository you can deploy the file there: mvn deploy:deploy-file -DgroupId=org.apache.hadoop -DartifactId=hadoop-hdfs -Dversion=0.22-SNAPSHOT -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id] Path to dependency: 1) org.apache.hadoop:hadoop-mapreduce-client-core:jar:1.0-SNAPSHOT 2) org.apache.hadoop:hadoop-hdfs:jar:0.22-SNAPSHOT ---------- 1 required artifact is missing. for artifact: org.apache.hadoop:hadoop-mapreduce-client-core:jar:1.0-SNAPSHOT from the specified remote repositories: central (http://repo1.maven.org/maven2), repository.jboss.org (http://repository.jboss.org/nexus/content/groups/public/)
        Hide
        Mahadev konar added a comment -

        sorry I should have mentioned I was running:

        mvn clean install assembly:assembly -Dmaven.test.skip.exec=true -P-cbuild
        
        Show
        Mahadev konar added a comment - sorry I should have mentioned I was running: mvn clean install assembly:assembly -Dmaven.test.skip.exec=true -P-cbuild
        Hide
        Luke Lu added a comment -

        Thanks for taking a look Mahadev.

        It's clear that the build is missing the required hdfs artifact. I recall that you were playing with hdfs build (that was broken) earlier, where you probably did a ant veryclean...

        Show
        Luke Lu added a comment - Thanks for taking a look Mahadev. It's clear that the build is missing the required hdfs artifact. I recall that you were playing with hdfs build (that was broken) earlier, where you probably did a ant veryclean...
        Hide
        Luke Lu added a comment -

        Mahadev, thanks for catching this. I introduced an incorrect default hdfs version (should be 0.22.0-SNAPSHOT instead of 0.22-SNAPSHOT) in hadoop-mapreduce-client-core's pom.xml. I didn't catch that because I was using a setting.xml with correct version of hdfs version defined.

        The v2 patch removed the superfluous default definition of the hdfs version property, which now should be inherited from parent poms.

        Show
        Luke Lu added a comment - Mahadev, thanks for catching this. I introduced an incorrect default hdfs version (should be 0.22.0-SNAPSHOT instead of 0.22-SNAPSHOT) in hadoop-mapreduce-client-core's pom.xml. I didn't catch that because I was using a setting.xml with correct version of hdfs version defined. The v2 patch removed the superfluous default definition of the hdfs version property, which now should be inherited from parent poms.
        Hide
        Luke Lu added a comment -

        v3 patch is rebased against the current HEAD and fixed a merge issue with Vinod's container monitoring commit.

        Show
        Luke Lu added a comment - v3 patch is rebased against the current HEAD and fixed a merge issue with Vinod's container monitoring commit.
        Hide
        Mahadev konar added a comment -

        I just committed this. thanks luke!

        Show
        Mahadev konar added a comment - I just committed this. thanks luke!

          People

          • Assignee:
            Luke Lu
            Reporter:
            Luke Lu
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development