Hadoop Common
  1. Hadoop Common
  2. HADOOP-7874

native libs should be under lib/native/ dir

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.1, 0.24.0
    • Fix Version/s: 0.23.1, 0.24.0
    • Component/s: build
    • Labels:
    • Hadoop Flags:
      Reviewed

      Description

      Currently common and hdfs SO files end up under lib/ dir with all JARs, they should end up under lib/native.

      In addition, the hadoop-config.sh script needs some cleanup when comes to native lib handling:

      • it is using lib/native/$ {JAVA_PLATFORM}

        for the java.library.path, when it should use lib/native.

      • it is looking for build/lib/native, this is from the old ant build, not applicable anymore.
      • it is looking for the libhdfs.a and adding to the java.librar.path, this is not correct.
      1. HADOOP-7874.patch
        2 kB
        Alejandro Abdelnur

        Issue Links

          Activity

          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12505695/HADOOP-7874.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/419//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12505695/HADOOP-7874.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/419//console This message is automatically generated.
          Hide
          Tom White added a comment -

          > it is using lib/native/$

          {JAVA_PLATFORM}

          for the java.library.path, when it should use lib/native

          Why has the arch directory been dropped?

          Show
          Tom White added a comment - > it is using lib/native/$ {JAVA_PLATFORM} for the java.library.path, when it should use lib/native Why has the arch directory been dropped?
          Hide
          Alejandro Abdelnur added a comment -

          The ARCH directory was dropped when we the tar layout for common/hdfs was introduced.

          While we could easily re-introduce it, IMO it is not the right thing to do for the following reasons:

          • The ARCH used by is a mix of the OS type (ie linux, mac,) and CPU ARCH (i386,amd64). However, having the same OS type and CPU arch does not ensure things will work because of native libraries depedencies required/available in different OSes (we can restrict ourselves to linux for practical purposes).
          • The build system does not handle building/packaging for different architectures.
          • Hadoop release distribution is a source TARBALL, you can easily build it correctly (talking about nativelibs here) in your target platform.
          Show
          Alejandro Abdelnur added a comment - The ARCH directory was dropped when we the tar layout for common/hdfs was introduced. While we could easily re-introduce it, IMO it is not the right thing to do for the following reasons: The ARCH used by is a mix of the OS type (ie linux, mac,) and CPU ARCH (i386,amd64). However, having the same OS type and CPU arch does not ensure things will work because of native libraries depedencies required/available in different OSes (we can restrict ourselves to linux for practical purposes). The build system does not handle building/packaging for different architectures. Hadoop release distribution is a source TARBALL, you can easily build it correctly (talking about nativelibs here) in your target platform.
          Hide
          Tom White added a comment -

          Sounds reasonable. What testing did you do on the current patch?

          Show
          Tom White added a comment - Sounds reasonable. What testing did you do on the current patch?
          Hide
          Alejandro Abdelnur added a comment -

          verified that the generate TAR has the native libs in the expected place lib/native/ an that hadoop-config.sh is setting the right path to java.library.path (by testing hadoop script)

          Show
          Alejandro Abdelnur added a comment - verified that the generate TAR has the native libs in the expected place lib/native/ an that hadoop-config.sh is setting the right path to java.library.path (by testing hadoop script)
          Hide
          Tom White added a comment -

          +1

          Show
          Tom White added a comment - +1
          Hide
          Alejandro Abdelnur added a comment -

          committed to trunk and branch-0.23

          Show
          Alejandro Abdelnur added a comment - committed to trunk and branch-0.23
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1428 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1428/)
          HADOOP-7874. native libs should be under lib/native/ dir. (tucu)

          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209286
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • /hadoop/common/trunk/hadoop-project-dist/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1428 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1428/ ) HADOOP-7874 . native libs should be under lib/native/ dir. (tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209286 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh /hadoop/common/trunk/hadoop-project-dist/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1354 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1354/)
          HADOOP-7874. native libs should be under lib/native/ dir. (tucu)

          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209286
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • /hadoop/common/trunk/hadoop-project-dist/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1354 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1354/ ) HADOOP-7874 . native libs should be under lib/native/ dir. (tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209286 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh /hadoop/common/trunk/hadoop-project-dist/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-0.23-Commit #245 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/245/)
          Merge -r 1209285:1209286 from trunk to branch. FIXES: HADOOP-7874

          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209289
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • /hadoop/common/branches/branch-0.23/hadoop-project-dist/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #245 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/245/ ) Merge -r 1209285:1209286 from trunk to branch. FIXES: HADOOP-7874 tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209289 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh /hadoop/common/branches/branch-0.23/hadoop-project-dist/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1379 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1379/)
          HADOOP-7874. native libs should be under lib/native/ dir. (tucu)

          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209286
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • /hadoop/common/trunk/hadoop-project-dist/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1379 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1379/ ) HADOOP-7874 . native libs should be under lib/native/ dir. (tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209286 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh /hadoop/common/trunk/hadoop-project-dist/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Commit #256 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/256/)
          Merge -r 1209285:1209286 from trunk to branch. FIXES: HADOOP-7874

          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209289
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • /hadoop/common/branches/branch-0.23/hadoop-project-dist/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #256 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/256/ ) Merge -r 1209285:1209286 from trunk to branch. FIXES: HADOOP-7874 tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209289 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh /hadoop/common/branches/branch-0.23/hadoop-project-dist/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #95 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/95/)
          Merge -r 1209285:1209286 from trunk to branch. FIXES: HADOOP-7874

          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209289
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • /hadoop/common/branches/branch-0.23/hadoop-project-dist/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #95 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/95/ ) Merge -r 1209285:1209286 from trunk to branch. FIXES: HADOOP-7874 tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209289 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh /hadoop/common/branches/branch-0.23/hadoop-project-dist/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-HAbranch-build #3 (See https://builds.apache.org/job/Hadoop-Hdfs-HAbranch-build/3/)
          Merge -r 1209285:1209286 from trunk to branch. FIXES: HADOOP-7874

          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209289
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • /hadoop/common/branches/branch-0.23/hadoop-project-dist/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-HAbranch-build #3 (See https://builds.apache.org/job/Hadoop-Hdfs-HAbranch-build/3/ ) Merge -r 1209285:1209286 from trunk to branch. FIXES: HADOOP-7874 tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209289 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh /hadoop/common/branches/branch-0.23/hadoop-project-dist/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #882 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/882/)
          HADOOP-7874. native libs should be under lib/native/ dir. (tucu)

          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209286
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • /hadoop/common/trunk/hadoop-project-dist/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #882 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/882/ ) HADOOP-7874 . native libs should be under lib/native/ dir. (tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209286 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh /hadoop/common/trunk/hadoop-project-dist/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Build #113 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/113/)
          Merge -r 1209285:1209286 from trunk to branch. FIXES: HADOOP-7874

          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209289
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • /hadoop/common/branches/branch-0.23/hadoop-project-dist/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #113 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/113/ ) Merge -r 1209285:1209286 from trunk to branch. FIXES: HADOOP-7874 tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209289 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh /hadoop/common/branches/branch-0.23/hadoop-project-dist/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #915 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/915/)
          HADOOP-7874. native libs should be under lib/native/ dir. (tucu)

          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209286
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • /hadoop/common/trunk/hadoop-project-dist/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #915 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/915/ ) HADOOP-7874 . native libs should be under lib/native/ dir. (tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209286 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh /hadoop/common/trunk/hadoop-project-dist/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Commit #241 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/241/)
          Merge -r 1209285:1209286 from trunk to branch. FIXES: HADOOP-7874

          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209289
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • /hadoop/common/branches/branch-0.23/hadoop-project-dist/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #241 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/241/ ) Merge -r 1209285:1209286 from trunk to branch. FIXES: HADOOP-7874 tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209289 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh /hadoop/common/branches/branch-0.23/hadoop-project-dist/pom.xml

            People

            • Assignee:
              Alejandro Abdelnur
              Reporter:
              Alejandro Abdelnur
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development