Hadoop Common
  1. Hadoop Common
  2. HADOOP-6173

src/native/packageNativeHadoop.sh only packages files with "hadoop" in the name

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: build, scripts
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      src/native/packageNativeHadoop.sh only packages files with "hadoop" in the name. This becomes too restrictive when a user wants to inject third-party native libraries into his/her own tar build.

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk #63 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/63/)
        . Change src/native/packageNativeHadoop.sh to package all native library files. Contributed by Hong Tang

        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk #63 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/63/ ) . Change src/native/packageNativeHadoop.sh to package all native library files. Contributed by Hong Tang
        Hide
        Tsz Wo Nicholas Sze added a comment -

        I have committed this. Thanks, Hong!

        Show
        Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Hong!
        Hide
        Hong Tang added a comment -

        Thanks Nicolas. I verified the patch through manual testing. Steps are listed as follows:

        Step 0: download hadoop-gpl-compression source repository and build jar files and native library.
        Step 1: copied hadoop-gpl-compression/build/hadoop-gpl-compression-0.1.0-dev/hadoop-gpl-compression-0.1.0-dev.jar => hadoop-common/lib; hadoop-gpl-compression/build/hadoop-gpl-compression-0.1.0-dev/lib/native/* => hadoop-common/lib/native
        Step 2: under hadoop-common, do "ant tar"
        Step 3: tar -tzf hadoop-common/build/hadoop-core-0.21.0-dev.tar.gz | grep libgplcompression => returning nothing
        Step 4: apply the patch, and redo steps 2-3, and grep shows libgplcompression.* are included in the tar

        Show
        Hong Tang added a comment - Thanks Nicolas. I verified the patch through manual testing. Steps are listed as follows: Step 0: download hadoop-gpl-compression source repository and build jar files and native library. Step 1: copied hadoop-gpl-compression/build/hadoop-gpl-compression-0.1.0-dev/hadoop-gpl-compression-0.1.0-dev.jar => hadoop-common/lib; hadoop-gpl-compression/build/hadoop-gpl-compression-0.1.0-dev/lib/native/* => hadoop-common/lib/native Step 2: under hadoop-common, do "ant tar" Step 3: tar -tzf hadoop-common/build/hadoop-core-0.21.0-dev.tar.gz | grep libgplcompression => returning nothing Step 4: apply the patch, and redo steps 2-3, and grep shows libgplcompression.* are included in the tar
        Hide
        Tsz Wo Nicholas Sze added a comment -

        +1 patch looks good

        > ... And Cos created a new issue HADOOP-6174 to track the unit tests for scripts. ...
        We are currently doing manual tests for scripts. Could you post the manual test results and "Also please list what manual steps were performed to verify this patch"?

        Show
        Tsz Wo Nicholas Sze added a comment - +1 patch looks good > ... And Cos created a new issue HADOOP-6174 to track the unit tests for scripts. ... We are currently doing manual tests for scripts. Could you post the manual test results and "Also please list what manual steps were performed to verify this patch"?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12415106/hadoop-6174-20090731.patch
        against trunk revision 804918.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/614/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/614/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/614/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/614/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12415106/hadoop-6174-20090731.patch against trunk revision 804918. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/614/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/614/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/614/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/614/console This message is automatically generated.
        Hide
        Hong Tang added a comment -

        Could we move forward with this jira?

        Show
        Hong Tang added a comment - Could we move forward with this jira?
        Hide
        Hong Tang added a comment -

        resubmit patch for hudson

        Show
        Hong Tang added a comment - resubmit patch for hudson
        Hide
        Hong Tang added a comment -

        BTW, the attached patch file name is misleading (I mistyped).

        Show
        Hong Tang added a comment - BTW, the attached patch file name is misleading (I mistyped).
        Hide
        Hong Tang added a comment -

        Looks like we need some basic support to test the scripts, particularly because the verification must be done after running "tar" or "package". Similar situation happens for HADOOP-6172. And Cos created a new issue HADOOP-6174 to track the unit tests for scripts. Could we resolve this issue for now?

        Show
        Hong Tang added a comment - Looks like we need some basic support to test the scripts, particularly because the verification must be done after running "tar" or "package". Similar situation happens for HADOOP-6172 . And Cos created a new issue HADOOP-6174 to track the unit tests for scripts. Could we resolve this issue for now?
        Hide
        Hong Tang added a comment -

        Just to clarify, the current build.xml behavior seems to be incorrect/inconsistent with what you described. If I put in a 3rd party jar in lib/, it will be copied to release tar, and if I put a 3rd party native library under lib/native with "hadoop" in their names (like libhadoopgplcompression.so), they will also be included.

        So I would argue that it is reasonable to include all jars and native libraries under lib/ to tar ball for two reasons:

        • it is a user's conscience decision to copy data under lib/ and thus the inclusion of these files is "by-choice".
        • the hadoop script currently includes all jars under lib/ in classpath, and all native libraries under lib/native/<arch>/ in sysproperty java.library.path. And it is reasonable for user to expect that if he/she runs an "ant tar" and untar the tarball somewhere else, it should behave exactly the same as the original place.

        I am fine to add a test to verify the behavior. How about just running md5sum over the set of files under lib and under package final destination?

        Show
        Hong Tang added a comment - Just to clarify, the current build.xml behavior seems to be incorrect/inconsistent with what you described. If I put in a 3rd party jar in lib/, it will be copied to release tar, and if I put a 3rd party native library under lib/native with "hadoop" in their names (like libhadoopgplcompression.so), they will also be included. So I would argue that it is reasonable to include all jars and native libraries under lib/ to tar ball for two reasons: it is a user's conscience decision to copy data under lib/ and thus the inclusion of these files is "by-choice". the hadoop script currently includes all jars under lib/ in classpath, and all native libraries under lib/native/<arch>/ in sysproperty java.library.path. And it is reasonable for user to expect that if he/she runs an "ant tar" and untar the tarball somewhere else, it should behave exactly the same as the original place. I am fine to add a test to verify the behavior. How about just running md5sum over the set of files under lib and under package final destination?
        Hide
        Doug Cutting added a comment -

        This seems to me more like a new feature request than a bug: existing code is designed to package what's to be included in a release and no more. You'd like to extend the package task to support inclusion of third party binaries. If we intend to maintain support for such a feature, should we add a test for it?

        Alternately you might be able to layer your own Ant build file that imports Hadoop's build.xml has a task that depends on the "package" target that then copies extra things into the dist directory, then a task that depends on that task and the "tar" task to bundle everything up.

        Show
        Doug Cutting added a comment - This seems to me more like a new feature request than a bug: existing code is designed to package what's to be included in a release and no more. You'd like to extend the package task to support inclusion of third party binaries. If we intend to maintain support for such a feature, should we add a test for it? Alternately you might be able to layer your own Ant build file that imports Hadoop's build.xml has a task that depends on the "package" target that then copies extra things into the dist directory, then a task that depends on that task and the "tar" task to bundle everything up.
        Hide
        Hong Tang added a comment -

        Trivial patch that relaxes the file name restriction.

        Show
        Hong Tang added a comment - Trivial patch that relaxes the file name restriction.
        Hide
        Hong Tang added a comment -

        The context of the problem is that I want to write a external script to package jars and native libraries from hadoop-gpl-compression to hadoop trunk tarball. The brief procedure is as follows:

        • build hadoop-gpl-compression, put jar file under $hadoop-src-dir/lib, and native libraries under $hadoop-src-dir/lib/native.
        • run "ant tar" under $hadoop-src-dir.

        However, in the tarball, hadoop-gpl-compression-0.1.0-dev.jar is included, but none of the native libraries are included (libgplcompression.*).

        Close examination reveals that it is because the script src/native/packageNativeHadoop.sh has the following lines:
        Line 45 $TAR hadoop | (cd $DIST_LIB_DIR/$platform/; $UNTAR)
        Line 61 $TAR hadoop | (cd $DIST_LIB_DIR/$platform/; $UNTAR)

        Show
        Hong Tang added a comment - The context of the problem is that I want to write a external script to package jars and native libraries from hadoop-gpl-compression to hadoop trunk tarball. The brief procedure is as follows: build hadoop-gpl-compression, put jar file under $hadoop-src-dir/lib, and native libraries under $hadoop-src-dir/lib/native. run "ant tar" under $hadoop-src-dir. However, in the tarball, hadoop-gpl-compression-0.1.0-dev.jar is included, but none of the native libraries are included (libgplcompression.*). Close examination reveals that it is because the script src/native/packageNativeHadoop.sh has the following lines: Line 45 $TAR hadoop | (cd $DIST_LIB_DIR/$platform/; $UNTAR) Line 61 $TAR hadoop | (cd $DIST_LIB_DIR/$platform/; $UNTAR)

          People

          • Assignee:
            Hong Tang
            Reporter:
            Hong Tang
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development