Hadoop Common
  1. Hadoop Common
  2. HADOOP-9911

hadoop 2.1.0-beta tarball only contains 32bit native libraries

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 2.1.0-beta, 2.2.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      I am setting up a cluster on 64bit linux and I noticed, that the tarball only ships with 32 bit libraries:

      $ pwd
      /opt/hadoop-2.1.0-beta/lib/native

      $ ls -al
      total 2376
      drwxr-xr-x 2 67974 users 4096 Aug 15 20:59 .
      drwxr-xr-x 3 67974 users 4096 Aug 15 20:59 ..
      rw-rr- 1 67974 users 598578 Aug 15 20:59 libhadoop.a
      rw-rr- 1 67974 users 764772 Aug 15 20:59 libhadooppipes.a
      lrwxrwxrwx 1 67974 users 18 Aug 15 20:59 libhadoop.so -> libhadoop.so.1.0.0
      -rwxr-xr-x 1 67974 users 407568 Aug 15 20:59 libhadoop.so.1.0.0
      rw-rr- 1 67974 users 304632 Aug 15 20:59 libhadooputils.a
      rw-rr- 1 67974 users 184414 Aug 15 20:59 libhdfs.a
      lrwxrwxrwx 1 67974 users 16 Aug 15 20:59 libhdfs.so -> libhdfs.so.0.0.0
      -rwxr-xr-x 1 67974 users 149556 Aug 15 20:59 libhdfs.so.0.0.0

      $ file *
      libhadoop.a: current ar archive
      libhadooppipes.a: current ar archive
      libhadoop.so: symbolic link to `libhadoop.so.1.0.0'
      libhadoop.so.1.0.0: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=0x527e88ec3e92a95389839bd3fc9d7dbdebf654d6, not stripped
      libhadooputils.a: current ar archive
      libhdfs.a: current ar archive
      libhdfs.so: symbolic link to `libhdfs.so.0.0.0'
      libhdfs.so.0.0.0: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=0xddb2abae9272f584edbe22c76b43fcda9436f685, not stripped

      $ hadoop checknative
      13/08/28 18:11:17 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      Native library checking:
      hadoop: false
      zlib: false
      snappy: false
      lz4: false
      bzip2: false
      13/08/28 18:11:17 INFO util.ExitUtil: Exiting with status 1

        Activity

        Hide
        Roman Shaposhnik added a comment -

        I guess a bigger question is whether it actually even makes sense to ship native bits at all. I'm afraid there's no way that Hadoop binary convenience artifacts can satisfy all the possible Linux combinations out there. 32bit vs. 64bit is just one aspect of it.

        Show
        Roman Shaposhnik added a comment - I guess a bigger question is whether it actually even makes sense to ship native bits at all. I'm afraid there's no way that Hadoop binary convenience artifacts can satisfy all the possible Linux combinations out there. 32bit vs. 64bit is just one aspect of it.
        Hide
        André Kelpe added a comment -

        The latest hadoop stable (1.2.1) is doing it. Why would hadoop 2.x not be able to do the same?

        Show
        André Kelpe added a comment - The latest hadoop stable (1.2.1) is doing it. Why would hadoop 2.x not be able to do the same?
        Hide
        Roman Shaposhnik added a comment -

        There was a change in layout in 2.x of where the native binaries are coming from – they are no longer under and arch-specific folder.

        Show
        Roman Shaposhnik added a comment - There was a change in layout in 2.x of where the native binaries are coming from – they are no longer under and arch-specific folder.
        Hide
        André Kelpe added a comment -

        I saw that the vote for hadoop 2.2 is open and this is still open. Is nobody using the native libs any longer or has this been overlooked?

        Show
        André Kelpe added a comment - I saw that the vote for hadoop 2.2 is open and this is still open. Is nobody using the native libs any longer or has this been overlooked?
        Hide
        André Kelpe added a comment -

        I just tried 2.2 and it is the same problem.

        Show
        André Kelpe added a comment - I just tried 2.2 and it is the same problem.
        Hide
        Rod added a comment -

        Voting for this because applications built around libhdfs depend on the binaries being available to end users. Missing 64bit binary lib would force users to build the hdfs project which might be simple for us, but it could be a deal breaker for end users.
        As it has been pointed out, the arch specific libs had been provided in past versions. Can we extend the "native" dir with the arch subfolders, and the appropriate libs? Thanks.

        Show
        Rod added a comment - Voting for this because applications built around libhdfs depend on the binaries being available to end users. Missing 64bit binary lib would force users to build the hdfs project which might be simple for us, but it could be a deal breaker for end users. As it has been pointed out, the arch specific libs had been provided in past versions. Can we extend the "native" dir with the arch subfolders, and the appropriate libs? Thanks.
        Hide
        Varun Dhussa added a comment -

        While migrating from 1.2 to 2.2, we ran into this. We are using a golang wrapper around libhdfs. So had to build the 64 bit binary ourselves. Would be great if it can be packaged.

        Show
        Varun Dhussa added a comment - While migrating from 1.2 to 2.2, we ran into this. We are using a golang wrapper around libhdfs. So had to build the 64 bit binary ourselves. Would be great if it can be packaged.
        Hide
        Sean Jensen-Grey added a comment -

        Why not ship the 64 bit library and leave the warning for the 32 bit users or ship both. But as it currently stands, shipping the 32 bit lib "breaks" a majority of the users.

        Show
        Sean Jensen-Grey added a comment - Why not ship the 64 bit library and leave the warning for the 32 bit users or ship both. But as it currently stands, shipping the 32 bit lib "breaks" a majority of the users.
        Hide
        André Kelpe added a comment -

        It seems that the current releases (2.5.2 and 2.6.0) are shipping with 64bit versions of the libraries. I guess this is related to changes in the build infrastructure or something. Whatever you do, please keep it this way.

        Show
        André Kelpe added a comment - It seems that the current releases (2.5.2 and 2.6.0) are shipping with 64bit versions of the libraries. I guess this is related to changes in the build infrastructure or something. Whatever you do, please keep it this way.
        Hide
        Rod added a comment -

        @André Kelpe are the 64bit versions packaged in the Apache release packages? I wasn't able to find them and was wondering where you found them. Thanks.

        Show
        Rod added a comment - @André Kelpe are the 64bit versions packaged in the Apache release packages? I wasn't able to find them and was wondering where you found them. Thanks.
        Hide
        André Kelpe added a comment -

        Rod yes, in 2.6.0 for instance I get this now:

         $ pwd
        /home/fs111/tools/hadoop-2.6.0/lib/native
        
        $ file *
        libhadoop.a:        current ar archive
        libhadooppipes.a:   current ar archive
        libhadoop.so:       symbolic link to `libhadoop.so.1.0.0'
        libhadoop.so.1.0.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0x2c42803ac908d7781a6c66a16723dd8ebb7dd76e, not stripped
        libhadooputils.a:   current ar archive
        libhdfs.a:          current ar archive
        libhdfs.so:         symbolic link to `libhdfs.so.0.0.0'
        libhdfs.so.0.0.0:   ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0xea14c6974f6b0f55237ec3bf233e9899750964b0, not stripped
        
        Show
        André Kelpe added a comment - Rod yes, in 2.6.0 for instance I get this now: $ pwd /home/fs111/tools/hadoop-2.6.0/lib/ native $ file * libhadoop.a: current ar archive libhadooppipes.a: current ar archive libhadoop.so: symbolic link to `libhadoop.so.1.0.0' libhadoop.so.1.0.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0x2c42803ac908d7781a6c66a16723dd8ebb7dd76e, not stripped libhadooputils.a: current ar archive libhdfs.a: current ar archive libhdfs.so: symbolic link to `libhdfs.so.0.0.0' libhdfs.so.0.0.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0xea14c6974f6b0f55237ec3bf233e9899750964b0, not stripped
        Hide
        Rod added a comment -

        Thanks @André Kelpe, I didn't see this change reported anywhere and would not have noticed if it weren't for your post.

        Show
        Rod added a comment - Thanks @André Kelpe, I didn't see this change reported anywhere and would not have noticed if it weren't for your post.
        Hide
        André Kelpe added a comment -

        It does not look like a deliberate change, more like a side effect caused by build-system changes. Let's hope it stays this way...

        Show
        André Kelpe added a comment - It does not look like a deliberate change, more like a side effect caused by build-system changes. Let's hope it stays this way...

          People

          • Assignee:
            Unassigned
            Reporter:
            André Kelpe
          • Votes:
            5 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:

              Development