Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: build, native
    • Labels:
      None
    • Environment:

      Ubuntu 10.10

    • Hadoop Flags:
      Reviewed

      Description

      My Hadoop installation is having trouble loading the native code library. It appears from the log below that java.library.path is missing the basedir in its path. The libraries are built, and present in the directory shown below (relative to hadoop-common directory). Instead of seeing:

      /build/native/Linux-amd64-64/lib

      I would expect to see:

      /path/to/hadoop-common/build/native/Linux-amd64-64/lib

      I'm working in branch-0.22.

      2011-01-10 17:09:27,695 DEBUG org.apache.hadoop.util.NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
      2011-01-10 17:09:27,695 DEBUG org.apache.hadoop.util.NativeCodeLoader: java.library.path=/build/native/Linux-amd64-64/lib
      2011-01-10 17:09:27,695 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

      1. HADOOP-7097.patch
        0.7 kB
        Noah Watkins

        Activity

        Hide
        Allen Wittenauer added a comment -

        Are you actually running it out of the build directory? If so, that's the problem.

        Show
        Allen Wittenauer added a comment - Are you actually running it out of the build directory? If so, that's the problem.
        Hide
        Noah Watkins added a comment -

        Yes, I'm running Hadoop from outside the source tree. However, in 0.21 and in trunk this issue doesn't seem to exist.

        Show
        Noah Watkins added a comment - Yes, I'm running Hadoop from outside the source tree. However, in 0.21 and in trunk this issue doesn't seem to exist.
        Hide
        Noah Watkins added a comment -

        I tracked down the problem in the trunk. It comes down to an old usage of HADOOP_CORE_HOME vs HADOOP_COMMON_HOME. That seems to fix it, and HADOOP_CORE_HOME isn't returned by any grep, so, is it special?

        diff --git a/bin/hadoop-config.sh b/bin/hadoop-config.sh
        index ee7b55c..4da14f8 100644
        — a/bin/hadoop-config.sh
        +++ b/bin/hadoop-config.sh
        @@ -192,9 +192,9 @@ if [ -d "$

        {HADOOP_COMMON_HOME}/build/native" -o -d "${HADOOP_COMMON_HOME}

        /lib/na

        if [ -d "$HADOOP_COMMON_HOME/build/native" ]; then
        if [ "x$JAVA_LIBRARY_PATH" != "x" ]; then

        • JAVA_LIBRARY_PATH=$ {JAVA_LIBRARY_PATH}:${HADOOP_CORE_HOME}/build/native/${JAVA_PLATFORM}/lib
          + JAVA_LIBRARY_PATH=${JAVA_LIBRARY_PATH}

          :$

          {HADOOP_COMMON_HOME}/build/native/${JAVA_PLATFORM}/lib
          else
          - JAVA_LIBRARY_PATH=${HADOOP_CORE_HOME}/build/native/${JAVA_PLATFORM}/lib
          + JAVA_LIBRARY_PATH=${HADOOP_COMMON_HOME}

          /build/native/$

          {JAVA_PLATFORM}

          /lib
          fi
          fi

        Show
        Noah Watkins added a comment - I tracked down the problem in the trunk. It comes down to an old usage of HADOOP_CORE_HOME vs HADOOP_COMMON_HOME. That seems to fix it, and HADOOP_CORE_HOME isn't returned by any grep, so, is it special? diff --git a/bin/hadoop-config.sh b/bin/hadoop-config.sh index ee7b55c..4da14f8 100644 — a/bin/hadoop-config.sh +++ b/bin/hadoop-config.sh @@ -192,9 +192,9 @@ if [ -d "$ {HADOOP_COMMON_HOME}/build/native" -o -d "${HADOOP_COMMON_HOME} /lib/na if [ -d "$HADOOP_COMMON_HOME/build/native" ]; then if [ "x$JAVA_LIBRARY_PATH" != "x" ]; then JAVA_LIBRARY_PATH=$ {JAVA_LIBRARY_PATH}:${HADOOP_CORE_HOME}/build/native/${JAVA_PLATFORM}/lib + JAVA_LIBRARY_PATH=${JAVA_LIBRARY_PATH} :$ {HADOOP_COMMON_HOME}/build/native/${JAVA_PLATFORM}/lib else - JAVA_LIBRARY_PATH=${HADOOP_CORE_HOME}/build/native/${JAVA_PLATFORM}/lib + JAVA_LIBRARY_PATH=${HADOOP_COMMON_HOME} /build/native/$ {JAVA_PLATFORM} /lib fi fi
        Hide
        Todd Lipcon added a comment -

        Would you mind uploading this as a patch attachment? JIRA screwed up the formatting a bit.

        Show
        Todd Lipcon added a comment - Would you mind uploading this as a patch attachment? JIRA screwed up the formatting a bit.
        Hide
        Noah Watkins added a comment -

        This seems to have fixed the problem I was seeing in the trunk.

        Show
        Noah Watkins added a comment - This seems to have fixed the problem I was seeing in the trunk.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12468111/HADOOP-7097.patch
        against trunk revision 1057970.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/176//testReport/
        Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/176//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/176//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12468111/HADOOP-7097.patch against trunk revision 1057970. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/176//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/176//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/176//console This message is automatically generated.
        Hide
        Todd Lipcon added a comment -

        +1

        Show
        Todd Lipcon added a comment - +1
        Hide
        Todd Lipcon added a comment -

        Committed to 0.22 and trunk. Thanks Noah!

        Show
        Todd Lipcon added a comment - Committed to 0.22 and trunk. Thanks Noah!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-22-branch #13 (See https://hudson.apache.org/hudson/job/Hadoop-Common-22-branch/13/)
        HADOOP-7097. JAVA_LIBRARY_PATH missing base directory. Contributed by Noah Watkins.

        Show
        Hudson added a comment - Integrated in Hadoop-Common-22-branch #13 (See https://hudson.apache.org/hudson/job/Hadoop-Common-22-branch/13/ ) HADOOP-7097 . JAVA_LIBRARY_PATH missing base directory. Contributed by Noah Watkins.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk #574 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/574/)
        HADOOP-7097. JAVA_LIBRARY_PATH missing base directory. Contributed by Noah Watkins.

        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk #574 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/574/ ) HADOOP-7097 . JAVA_LIBRARY_PATH missing base directory. Contributed by Noah Watkins.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #479 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/479/)

        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #479 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/479/ )

          People

          • Assignee:
            Noah Watkins
            Reporter:
            Noah Watkins
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development