Hadoop Common
  1. Hadoop Common
  2. HADOOP-7737

normalize hadoop-mapreduce & hadoop-dist dist/tar build with common/hdfs

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0, 0.24.0
    • Fix Version/s: 0.23.0, 0.24.0
    • Component/s: build
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Normalize the build fo hadoop-mapreduce and hadoop-dist with hadoop-common and hadoop-hdfs making the -Pdist and -Dtar maven options to be consistent.

      • -Pdist should create the layout
      • -Dtar should create the TAR
      1. HADOOP-7737.patch
        9 kB
        Alejandro Abdelnur
      2. HADOOP-7737.patch
        9 kB
        Alejandro Abdelnur

        Issue Links

          Activity

          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12498817/HADOOP-7737.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/290//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12498817/HADOOP-7737.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/290//console This message is automatically generated.
          Hide
          Alejandro Abdelnur added a comment -

          Updated patch only creates the hadoop-dist TAR if the -Dtar option is used.

          Show
          Alejandro Abdelnur added a comment - Updated patch only creates the hadoop-dist TAR if the -Dtar option is used.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12498842/HADOOP-7737.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/291//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12498842/HADOOP-7737.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/291//console This message is automatically generated.
          Hide
          Tom White added a comment -

          +1 I tried with and without -Dtar. I managed to run a MR job from the "exploded" directory.

          export HADOOP_COMMON_HOME=$(pwd)/$(ls -d hadoop-common-project/hadoop-common/target/hadoop-common-*-SNAPSHOT)
          export HADOOP_HDFS_HOME=$(pwd)/$(ls -d hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-*-SNAPSHOT)
          export HADOOP_MAPRED_HOME=$(pwd)/$(ls -d hadoop-mapreduce-project/target/hadoop-mapreduce-*-SNAPSHOT)
          export YARN_HOME=$HADOOP_MAPRED_HOME
          
          export PATH=$HADOOP_COMMON_HOME/bin:$HADOOP_HDFS_HOME/bin:$HADOOP_MAPRED_HOME/bin:$PATH
          
          cat > $YARN_HOME/conf/yarn-site.xml  << EOF
          <?xml version="1.0"?>
          <configuration>
              <!-- Site specific YARN configuration properties -->
              <property>
                <name>yarn.nodemanager.aux-services</name>
                <value>mapreduce.shuffle</value>
              </property>
              <property>
                <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
                <value>org.apache.hadoop.mapred.ShuffleHandler</value>
              </property>
          </configuration>
          EOF
          
          cd hadoop-mapreduce-project
          ant examples -Dresolvers=internal
          cd ..
          export HADOOP_CLASSPATH=$YARN_HOME/modules/*
          mkdir in
          cp BUILDING.txt in/
          hadoop jar hadoop-mapreduce-project/build/hadoop-mapreduce-examples-0.24.0-SNAPSHOT.jar wordcount -Dmapreduce.job.user.name=$USER in out
          

          I'll add this to http://wiki.apache.org/hadoop/HowToSetupYourDevelopmentEnvironment after it's committed.

          Show
          Tom White added a comment - +1 I tried with and without -Dtar. I managed to run a MR job from the "exploded" directory. export HADOOP_COMMON_HOME=$(pwd)/$(ls -d hadoop-common-project/hadoop-common/target/hadoop-common-*-SNAPSHOT) export HADOOP_HDFS_HOME=$(pwd)/$(ls -d hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-*-SNAPSHOT) export HADOOP_MAPRED_HOME=$(pwd)/$(ls -d hadoop-mapreduce-project/target/hadoop-mapreduce-*-SNAPSHOT) export YARN_HOME=$HADOOP_MAPRED_HOME export PATH=$HADOOP_COMMON_HOME/bin:$HADOOP_HDFS_HOME/bin:$HADOOP_MAPRED_HOME/bin:$PATH cat > $YARN_HOME/conf/yarn-site.xml << EOF <?xml version="1.0"?> <configuration> <!-- Site specific YARN configuration properties --> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce.shuffle</value> </property> <property> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> <value>org.apache.hadoop.mapred.ShuffleHandler</value> </property> </configuration> EOF cd hadoop-mapreduce-project ant examples -Dresolvers=internal cd .. export HADOOP_CLASSPATH=$YARN_HOME/modules/* mkdir in cp BUILDING.txt in/ hadoop jar hadoop-mapreduce-project/build/hadoop-mapreduce-examples-0.24.0-SNAPSHOT.jar wordcount -Dmapreduce.job.user.name=$USER in out I'll add this to http://wiki.apache.org/hadoop/HowToSetupYourDevelopmentEnvironment after it's committed.
          Hide
          Alejandro Abdelnur added a comment -

          I've just committed this to trunk (will commit to 0.23 later with other mvn-related changes). Thanks, Alejandro.

          Show
          Alejandro Abdelnur added a comment - I've just committed this to trunk (will commit to 0.23 later with other mvn-related changes). Thanks, Alejandro.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1072 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1072/)
          HADOOP-7737. normalize hadoop-mapreduce & hadoop-dist dist/tar build with common/hdfs (tucu)

          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1183068
          Files :

          • /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-dist/pom.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1072 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1072/ ) HADOOP-7737 . normalize hadoop-mapreduce & hadoop-dist dist/tar build with common/hdfs (tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1183068 Files : /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-dist/pom.xml /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1150 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1150/)
          HADOOP-7737. normalize hadoop-mapreduce & hadoop-dist dist/tar build with common/hdfs (tucu)

          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1183068
          Files :

          • /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-dist/pom.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1150 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1150/ ) HADOOP-7737 . normalize hadoop-mapreduce & hadoop-dist dist/tar build with common/hdfs (tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1183068 Files : /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-dist/pom.xml /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1091 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1091/)
          HADOOP-7737. normalize hadoop-mapreduce & hadoop-dist dist/tar build with common/hdfs (tucu)

          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1183068
          Files :

          • /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-dist/pom.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1091 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1091/ ) HADOOP-7737 . normalize hadoop-mapreduce & hadoop-dist dist/tar build with common/hdfs (tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1183068 Files : /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-dist/pom.xml /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #830 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/830/)
          HADOOP-7737. normalize hadoop-mapreduce & hadoop-dist dist/tar build with common/hdfs (tucu)

          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1183068
          Files :

          • /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-dist/pom.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #830 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/830/ ) HADOOP-7737 . normalize hadoop-mapreduce & hadoop-dist dist/tar build with common/hdfs (tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1183068 Files : /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-dist/pom.xml /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #860 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/860/)
          HADOOP-7737. normalize hadoop-mapreduce & hadoop-dist dist/tar build with common/hdfs (tucu)

          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1183068
          Files :

          • /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-dist/pom.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #860 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/860/ ) HADOOP-7737 . normalize hadoop-mapreduce & hadoop-dist dist/tar build with common/hdfs (tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1183068 Files : /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-dist/pom.xml /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Commit #105 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/105/)
          Merge -r 1183067:1183068 from trunk to branch. FIXES: HADOOP-7737

          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190632
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-dist/pom.xml
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #105 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/105/ ) Merge -r 1183067:1183068 from trunk to branch. FIXES: HADOOP-7737 tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190632 Files : /hadoop/common/branches/branch-0.23/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-dist/pom.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Commit #100 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/100/)
          Merge -r 1183067:1183068 from trunk to branch. FIXES: HADOOP-7737

          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190632
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-dist/pom.xml
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #100 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/100/ ) Merge -r 1183067:1183068 from trunk to branch. FIXES: HADOOP-7737 tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190632 Files : /hadoop/common/branches/branch-0.23/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-dist/pom.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-0.23-Commit #100 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/100/)
          Merge -r 1183067:1183068 from trunk to branch. FIXES: HADOOP-7737

          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190632
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-dist/pom.xml
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #100 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/100/ ) Merge -r 1183067:1183068 from trunk to branch. FIXES: HADOOP-7737 tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190632 Files : /hadoop/common/branches/branch-0.23/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-dist/pom.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #54 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/54/)
          Merge -r 1183067:1183068 from trunk to branch. FIXES: HADOOP-7737

          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190632
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-dist/pom.xml
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #54 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/54/ ) Merge -r 1183067:1183068 from trunk to branch. FIXES: HADOOP-7737 tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190632 Files : /hadoop/common/branches/branch-0.23/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-dist/pom.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Build #72 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/72/)
          Merge -r 1183067:1183068 from trunk to branch. FIXES: HADOOP-7737

          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190632
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-dist/pom.xml
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #72 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/72/ ) Merge -r 1183067:1183068 from trunk to branch. FIXES: HADOOP-7737 tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190632 Files : /hadoop/common/branches/branch-0.23/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-dist/pom.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/pom.xml

            People

            • Assignee:
              Alejandro Abdelnur
              Reporter:
              Alejandro Abdelnur
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development