Hadoop Common
  1. Hadoop Common
  2. HADOOP-6342

Create a script to squash a common, hdfs, and mapreduce tarball into a single hadoop tarball

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Not a Problem
    • Affects Version/s: None
    • Fix Version/s: 0.22.0
    • Component/s: build
    • Labels:
      None

      Description

      It would be convenient for the transition if we had a script to take a set of common, hdfs, and mapreduce tarballs and merge them into a single tarball. This is intended just to help users who don't want to transition to split projects for deployment immediately.

      1. HADOOP-6342.patch
        16 kB
        Aaron Kimball
      2. HADOOP-6342.2.patch
        16 kB
        Aaron Kimball
      3. h-6342.patch
        2 kB
        Owen O'Malley
      4. tar-munge
        2 kB
        Owen O'Malley
      5. tar-munge
        2 kB
        Tom White

        Issue Links

          Activity

          Hide
          Harsh J added a comment -

          Looks like this was not marked as resolved post Tom's comment earlier, of https://issues.apache.org/jira/browse/HADOOP-6846 having fixed it in 0.22

          Show
          Harsh J added a comment - Looks like this was not marked as resolved post Tom's comment earlier, of https://issues.apache.org/jira/browse/HADOOP-6846 having fixed it in 0.22
          Show
          Tom White added a comment - We can close this now as it is subsumed by HADOOP-6846 (see https://issues.apache.org/jira/browse/HADOOP-6846?focusedCommentId=13001571&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13001571 ).
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12446199/tar-munge
          against trunk revision 1071364.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/284//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446199/tar-munge against trunk revision 1071364. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/284//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12446199/tar-munge
          against trunk revision 1031422.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/66//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446199/tar-munge against trunk revision 1031422. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/66//console This message is automatically generated.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Looks like this is missed in the 0.21.0 release.

          Show
          Vinod Kumar Vavilapalli added a comment - Looks like this is missed in the 0.21.0 release.
          Hide
          Tom White added a comment -

          > This should probably only be committed to 0.21 as a compatibility hack.

          Good idea - I'll commit this to the 0.21 branch in src (where there are a couple of other dev scripts).

          Show
          Tom White added a comment - > This should probably only be committed to 0.21 as a compatibility hack. Good idea - I'll commit this to the 0.21 branch in src (where there are a couple of other dev scripts).
          Hide
          Tom White added a comment -

          Modified version of tar-munge that works with the changes in HADOOP-6794 (and related JIRAs) to split configuration and scripts. Also updates "core" to "common". Also a few other minor changes to make tarball more similar to a 0.20 series tarball.

          Show
          Tom White added a comment - Modified version of tar-munge that works with the changes in HADOOP-6794 (and related JIRAs) to split configuration and scripts. Also updates "core" to "common". Also a few other minor changes to make tarball more similar to a 0.20 series tarball.
          Hide
          Owen O'Malley added a comment -

          This version removes the restriction on where the tarballs are and that they have the same directory. It also creates symlinks for the jars without their version numbers.

          Show
          Owen O'Malley added a comment - This version removes the restriction on where the tarballs are and that they have the same directory. It also creates symlinks for the jars without their version numbers.
          Hide
          Owen O'Malley added a comment -

          Rather than go through a make file, this script just takes three tarballs and puts them together.

          This should probably only be committed to 0.21 as a compatibility hack.

          Show
          Owen O'Malley added a comment - Rather than go through a make file, this script just takes three tarballs and puts them together. This should probably only be committed to 0.21 as a compatibility hack.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12423725/HADOOP-6342.2.patch
          against trunk revision 831416.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/120/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/120/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/120/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/120/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12423725/HADOOP-6342.2.patch against trunk revision 831416. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/120/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/120/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/120/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/120/console This message is automatically generated.
          Hide
          Aaron Kimball added a comment -

          new patch, fixed release audit warning in the .gitignore file it adds.

          Show
          Aaron Kimball added a comment - new patch, fixed release audit warning in the .gitignore file it adds.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12423722/HADOOP-6342.patch
          against trunk revision 831416.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings).

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/119/testReport/
          Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/119/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/119/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/119/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/119/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12423722/HADOOP-6342.patch against trunk revision 831416. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings). +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/119/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/119/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/119/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/119/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/119/console This message is automatically generated.
          Hide
          Aaron Kimball added a comment -

          This patch provides a Makefile that can build all three projects and merge them. It's based on a script that I have been using myself that I finally got around to cleaning up a bit.

          This adds a Makefile in src/buildscripts/ as well as some shell scripts that it calls out to.

          Running make combined-tar will create a combined tarball. It can also execute other commands across all three projects, e.g., make compile-core and combine their results into a single build directory. See the included README and the Makefile's comments for the full story.

          In order to support mapred, hdfs, and core running in the same directory, this also modifies bin/hadoop-config.sh to assume that $HADOOP_CORE_HOME can satisify $HADOOP_MAPRED_HOME and $HADOOP_HDFS_HOME in the absense of hdfs/ and mapred/ subdirs.

          Finally, this current version contains a hack in combine-bindirs.sh to reconcile the competing versions of various libraries used by mapred, hdfs, and common. These should be reconciled via their ivy.xml files – separate ticket? They are left in for now for demonstration that this Makefile system works.

          I've tested this by running the various commands and verifying they work. I also used 'make combined-tar' to build a tarball, unzipped it elsewhere, and used it to start a pseudodistributed instance and run pi. I also ran this from my hadoop-common/build/combined-hadoop directory created with make combined-binary.

          If you commit this or something like it, you'll need to svn add src/buildscripts/

          Show
          Aaron Kimball added a comment - This patch provides a Makefile that can build all three projects and merge them. It's based on a script that I have been using myself that I finally got around to cleaning up a bit. This adds a Makefile in src/buildscripts/ as well as some shell scripts that it calls out to. Running make combined-tar will create a combined tarball. It can also execute other commands across all three projects, e.g., make compile-core and combine their results into a single build directory. See the included README and the Makefile's comments for the full story. In order to support mapred, hdfs, and core running in the same directory, this also modifies bin/hadoop-config.sh to assume that $HADOOP_CORE_HOME can satisify $HADOOP_MAPRED_HOME and $HADOOP_HDFS_HOME in the absense of hdfs/ and mapred/ subdirs. Finally, this current version contains a hack in combine-bindirs.sh to reconcile the competing versions of various libraries used by mapred, hdfs, and common. These should be reconciled via their ivy.xml files – separate ticket? They are left in for now for demonstration that this Makefile system works. I've tested this by running the various commands and verifying they work. I also used 'make combined-tar' to build a tarball, unzipped it elsewhere, and used it to start a pseudodistributed instance and run pi. I also ran this from my hadoop-common/build/combined-hadoop directory created with make combined-binary . If you commit this or something like it, you'll need to svn add src/buildscripts/

            People

            • Assignee:
              Owen O'Malley
              Reporter:
              Owen O'Malley
            • Votes:
              1 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development