Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-7825

Hadoop wrapper script not picking up native libs correctly

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 0.20.205.0
    • Fix Version/s: None
    • Component/s: scripts
    • Labels:
      None
    • Environment:

      Debian 6.0 x64_64
      java version "1.6.0_26"
      Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
      Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)

      Description

      Originally discussed in https://mail-archives.apache.org/mod_mbox/hadoop-common-user/201111.mbox/%3C4EC3A3AE.7060402%40deri.org%3E

      I'm testing out native lib support on our test amd64 test cluster
      running 0.20.205 running the following

      ./bin/hadoop jar hadoop-test-0.20.205.0.jar testsequencefile -seed 0
      -count 1000 -compressType RECORD xxx -codec
      org.apache.hadoop.io.compress.GzipCodec -check 2

      it fails with

      WARN util.NativeCodeLoader: Unable to load native-hadoop library for
      your platform... using builtin-java classes where applicable

      Looking at

      bin/hadoop

      it seems to successfully detect that the native libs are available (they
      seem to come pre-compiled with 0.20.205 which is nice)

      if [ -d "${HADOOP_HOME}/lib/native" ]; then
      if [ "x$JAVA_LIBRARY_PATH" != "x" ]; then

      JAVA_LIBRARY_PATH=${JAVA_LIBRARY_PATH}:${HADOOP_HOME}/lib/native/${JAVA_PLATFORM}
      else
      JAVA_LIBRARY_PATH=${HADOOP_HOME}/lib/native/${JAVA_PLATFORM}
      fi
      fi

      and sets JAVA_LIBRARY_PATH to contain them.

      Then in the following line, if ${HADOOP_HOME}/lib contains libhadoop.a
      (which is seems to in the stock tar) then it proceeds to ignore the
      native libs

      if [ -e "${HADOOP_PREFIX}/lib/libhadoop.a" ]; then
      JAVA_LIBRARY_PATH=${HADOOP_PREFIX}/lib
      fi

      The libhadoop.a in ${HADOOP_HOME}/lib seems to be a copy of the lib/native/Linux-i386-32 going from the filesizes (and also noted by https://mail-archives.apache.org/mod_mbox/hadoop-common-user/201111.mbox/%3CCAOcnVr2AzUDnN0LFhmTqUmAyujYtvhfkmMm_J0R-bmxw2wU+9A@mail.gmail.com%3E)

      hadoop@testhbase01:~$ ls -la hadoop/lib/libhadoop.*
      rw-rr- 1 hadoop hadoop 237244 Oct 7 08:20 hadoop/lib/libhadoop.a
      rw-rr- 1 hadoop hadoop 877 Oct 7 08:20 hadoop/lib/libhadoop.la
      rw-rr- 1 hadoop hadoop 160438 Oct 7 08:20 hadoop/lib/libhadoop.so
      rw-rr- 1 hadoop hadoop 160438 Oct 7 08:19 hadoop/lib/libhadoop.so.1.0.0
      hadoop@testhbase01:~$ ls -la hadoop/lib/native/Linux-i386-32/
      total 728
      drwxr-xr-x 3 hadoop hadoop 4096 Nov 15 14:05 .
      drwxr-xr-x 5 hadoop hadoop 4096 Oct 7 08:24 ..
      rw-rr- 1 hadoop hadoop 237244 Oct 7 08:20 libhadoop.a
      rw-rr- 1 hadoop hadoop 877 Oct 7 08:20 libhadoop.la
      rw-rr- 1 hadoop hadoop 160438 Oct 7 08:20 libhadoop.so
      rw-rr- 1 hadoop hadoop 160438 Oct 7 08:20 libhadoop.so.1
      rw-rr- 1 hadoop hadoop 160438 Oct 7 08:20 libhadoop.so.1.0.0

      A possible solution includes removing libhadoop.a and friends from ${HADOOP_HOME}/lib and possibly also modifying the hadoop wrapper to remove

      if [ -e "${HADOOP_PREFIX}/lib/libhadoop.a" ]; then
      JAVA_LIBRARY_PATH=${HADOOP_PREFIX}/lib
      fi

      unless there is some other reason for this to exist.

      This was also noted in a comment to HADOOP-6453

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                smulcahy stephen mulcahy
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: