Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-2009

configure script for compiling hadoop native doesn't set lzo lib name correctly

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 0.13.1
    • Fix Version/s: None
    • Component/s: build
    • Labels:
      None
    • Environment:

      Fedora, amd64

      Description

      Looks like this was already reported (but not resolved on the the list): http://tinyurl.com/2rwu6x

      I would like to compile libhadoop on amd64/Fedora and everything seems kosher until I hit this compile error:

      [exec] /home/jssarma/fbprojects/hadoop-0.13.1/src/native/src/org/apache/hadoop/io/compress/lzo/LzoCompressor.c:116: error: syntax error before ',' token

      the line in question is:

      // Load liblzo2.so
      liblzo2 = dlopen(HADOOP_LZO_LIBRARY, RTLD_LAZY | RTLD_GLOBAL);

      seems like this is being set by:

      configure:#define HADOOP_LZO_LIBRARY $

      {ac_cv_libname_lzo2}

      I tried executing the relevant part of configure by hand:

      if test -z "`$

      {CC}

      -o conftest conftest.c -llzo2 2>&1`"; then
      if test ! -z "`which objdump`"; then
      ac_cv_libname_lzo2="`objdump -p conftest | grep NEEDED | grep lzo2 | sed 's/\W*NEEDED\W*(.)\W$/\
      \"\1\"/'`"

      This is not working on my system, since:
      > objdump -p conftest | grep NEEDED
      NEEDED libc.so.6

      So that would explain the compile error. Editing the configure script manually for now works.

        Activity

        Hide
        acmurthy Arun C Murthy added a comment -

        Joydeep,

        I don't have access to an amd64-fedora box, could you please try this and let me know?

        $ echo 'int main(int argc, char **argv){return 0;}' > conftest.c
        $ gcc -o conftest conftest.c -llzo2
        $ objdump -p conftest | grep NEEDED
        

        on my system I get:

        $ objdump -p conftest | grep NEEDED
          NEEDED      liblzo2.so.2
          NEEDED      libc.so.6
        
        Show
        acmurthy Arun C Murthy added a comment - Joydeep, I don't have access to an amd64-fedora box, could you please try this and let me know? $ echo 'int main(int argc, char **argv){return 0;}' > conftest.c $ gcc -o conftest conftest.c -llzo2 $ objdump -p conftest | grep NEEDED on my system I get: $ objdump -p conftest | grep NEEDED NEEDED liblzo2.so.2 NEEDED libc.so.6
        Hide
        jsensarma Joydeep Sen Sarma added a comment -

        as i reported - i have already tried this out (retried just in case). On my platform - i only see libc.so.6 as the required library.

        I am wondering if it is because i don't have .so versions of lz libraries - only .a:
        [jssarma@dev046 if]$ ls -l /usr/local/lib/lzo2
        rw-rr- 1 root root 847206 Feb 28 2007 /usr/local/lib/liblzo2.a
        -rwxr-xr-x 1 root root 752 Feb 28 2007 /usr/local/lib/liblzo2.la

        Show
        jsensarma Joydeep Sen Sarma added a comment - as i reported - i have already tried this out (retried just in case). On my platform - i only see libc.so.6 as the required library. I am wondering if it is because i don't have .so versions of lz libraries - only .a: [jssarma@dev046 if] $ ls -l /usr/local/lib/ lzo2 rw-r r - 1 root root 847206 Feb 28 2007 /usr/local/lib/liblzo2.a -rwxr-xr-x 1 root root 752 Feb 28 2007 /usr/local/lib/liblzo2.la
        Hide
        acmurthy Arun C Murthy added a comment -

        Yes, that is weird. The rpm which got you lzo2 shud come with both dynamic and static versions of lzo... could you check and maybe reinstall lzo? Thanks.

        Show
        acmurthy Arun C Murthy added a comment - Yes, that is weird. The rpm which got you lzo2 shud come with both dynamic and static versions of lzo... could you check and maybe reinstall lzo? Thanks.
        Hide
        jsensarma Joydeep Sen Sarma added a comment -

        As I mentioned - i already have a workaround - i edited configure to hard code the library path.

        I don't know how lzo was installed - the system was imaged by IT folks and it is the way it is. The issue is how to help the next person who hits this error. Clearly, if grep NEEDED returns empty - there's an error - and either the script/build needs to stop. Perhaps you might want to just print out an error from configure and ask them to change the script. or get it off a environment variable.

        Show
        jsensarma Joydeep Sen Sarma added a comment - As I mentioned - i already have a workaround - i edited configure to hard code the library path. I don't know how lzo was installed - the system was imaged by IT folks and it is the way it is. The issue is how to help the next person who hits this error. Clearly, if grep NEEDED returns empty - there's an error - and either the script/build needs to stop. Perhaps you might want to just print out an error from configure and ask them to change the script. or get it off a environment variable.
        Hide
        acmurthy Arun C Murthy added a comment -

        The idea of using gnu's auto* toolset is to not rely on hard-coded paths since they are system/distribution specific.

        So, well, any information you provide does help in solving the problem for the next user. Clearly it is easier to fix if you could confirm that the compile went through with liblzo2.so being present on your system, else there might be other doors to knock-on.

        Btw, could you also attach the output of the entire compilation process i.e. the parts where configure checks for zlib/lzo headers and so on?

        Show
        acmurthy Arun C Murthy added a comment - The idea of using gnu's auto* toolset is to not rely on hard-coded paths since they are system/distribution specific. So, well, any information you provide does help in solving the problem for the next user. Clearly it is easier to fix if you could confirm that the compile went through with liblzo2.so being present on your system, else there might be other doors to knock-on. Btw, could you also attach the output of the entire compilation process i.e. the parts where configure checks for zlib/lzo headers and so on?
        Hide
        jsensarma Joydeep Sen Sarma added a comment -

        here's the log. i think that the error should be detected in configure and auto-configuration should either fail outright or print a loud warning that compilation of native libs will not work unless some steps are taken. It might take me some more time to test with .so

        [exec] if /bin/sh ../../../../../../../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I.\
        -I/home/jssarma/fbprojects/hadoop-0.13.1/src/native/src/org/apache/hadoop/io/compress/lzo -I../../\
        ../../../../.. -I/usr/local/java/include -I/usr/local/java/include/linux -I/home/jssarma/fbproject\
        s/hadoop-0.13.1/src/native/src -g -Wall -fPIC -O2 -m64 -g -O2 -MT LzoCompressor.lo -MD -MP -MF ".d\
        eps/LzoCompressor.Tpo" -c -o LzoCompressor.lo /home/jssarma/fbprojects/hadoop-0.13.1/src/native/src\
        /org/apache/hadoop/io/compress/lzo/LzoCompressor.c; \
        [exec] then mv -f ".deps/LzoCompressor.Tpo" ".deps/LzoCompressor.Plo"; else rm -f ".deps/LzoCo\
        mpressor.Tpo"; exit 1; fi
        [exec] gcc -DHAVE_CONFIG_H -I. -I/home/jssarma/fbprojects/hadoop-0.13.1/src/native/src/org/ap\
        ache/hadoop/io/compress/lzo -I../../../../../../.. -I/usr/local/java/include -I/usr/local/java/incl\
        ude/linux -I/home/jssarma/fbprojects/hadoop-0.13.1/src/native/src -g -Wall -fPIC -O2 -m64 -g -O2 -M\
        T LzoCompressor.lo -MD -MP -MF .deps/LzoCompressor.Tpo -c /home/jssarma/fbprojects/hadoop-0.13.1/sr\
        c/native/src/org/apache/hadoop/io/compress/lzo/LzoCompressor.c -fPIC -DPIC -o .libs/LzoCompressor.\
        o
        [exec] /home/jssarma/fbprojects/hadoop-0.13.1/src/native/src/org/apache/hadoop/io/compress/lzo\
        /LzoCompressor.c: In function 'Java_org_apache_hadoop_io_compress_lzo_LzoCompressor_initIDs':
        [exec] /home/jssarma/fbprojects/hadoop-0.13.1/src/native/src/org/apache/hadoop/io/compress/lzo\
        /LzoCompressor.c:116: error: syntax error before '/' token
        [exec] make[2]: Leaving directory `/home/jssarma/fbprojects/hadoop-0.13.1/build/native/Linux-a\
        md64-64/src/org/apache/hadoop/io/compress/lzo'
        [exec] make[2]: *** [LzoCompressor.lo] Error 1
        [exec] make[1]: Leaving directory `/home/jssarma/fbprojects/hadoop-0.13.1/build/native/Linux-a\
        md64-64'
        [exec] make[1]: *** [all-recursive] Error 1
        [exec] make: *** [all] Error 2

        BUILD FAILED
        /home/jssarma/fbprojects/hadoop-0.13.1/build.xml:285: exec returned: 2
        at org.apache.tools.ant.taskdefs.ExecTask.runExecute(ExecTask.java:591)
        at org.apache.tools.ant.taskdefs.ExecTask.runExec(ExecTask.java:617)
        at org.apache.tools.ant.taskdefs.ExecTask.execute(ExecTask.java:452)
        at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:275)

        Show
        jsensarma Joydeep Sen Sarma added a comment - here's the log. i think that the error should be detected in configure and auto-configuration should either fail outright or print a loud warning that compilation of native libs will not work unless some steps are taken. It might take me some more time to test with .so [exec] if /bin/sh ../../../../../../../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I.\ -I/home/jssarma/fbprojects/hadoop-0.13.1/src/native/src/org/apache/hadoop/io/compress/lzo -I../../\ ../../../../.. -I/usr/local/java/include -I/usr/local/java/include/linux -I/home/jssarma/fbproject\ s/hadoop-0.13.1/src/native/src -g -Wall -fPIC -O2 -m64 -g -O2 -MT LzoCompressor.lo -MD -MP -MF ".d\ eps/LzoCompressor.Tpo" -c -o LzoCompressor.lo /home/jssarma/fbprojects/hadoop-0.13.1/src/native/src\ /org/apache/hadoop/io/compress/lzo/LzoCompressor.c; \ [exec] then mv -f ".deps/LzoCompressor.Tpo" ".deps/LzoCompressor.Plo"; else rm -f ".deps/LzoCo\ mpressor.Tpo"; exit 1; fi [exec] gcc -DHAVE_CONFIG_H -I. -I/home/jssarma/fbprojects/hadoop-0.13.1/src/native/src/org/ap\ ache/hadoop/io/compress/lzo -I../../../../../../.. -I/usr/local/java/include -I/usr/local/java/incl\ ude/linux -I/home/jssarma/fbprojects/hadoop-0.13.1/src/native/src -g -Wall -fPIC -O2 -m64 -g -O2 -M\ T LzoCompressor.lo -MD -MP -MF .deps/LzoCompressor.Tpo -c /home/jssarma/fbprojects/hadoop-0.13.1/sr\ c/native/src/org/apache/hadoop/io/compress/lzo/LzoCompressor.c -fPIC -DPIC -o .libs/LzoCompressor.\ o [exec] /home/jssarma/fbprojects/hadoop-0.13.1/src/native/src/org/apache/hadoop/io/compress/lzo\ /LzoCompressor.c: In function 'Java_org_apache_hadoop_io_compress_lzo_LzoCompressor_initIDs': [exec] /home/jssarma/fbprojects/hadoop-0.13.1/src/native/src/org/apache/hadoop/io/compress/lzo\ /LzoCompressor.c:116: error: syntax error before '/' token [exec] make [2] : Leaving directory `/home/jssarma/fbprojects/hadoop-0.13.1/build/native/Linux-a\ md64-64/src/org/apache/hadoop/io/compress/lzo' [exec] make [2] : *** [LzoCompressor.lo] Error 1 [exec] make [1] : Leaving directory `/home/jssarma/fbprojects/hadoop-0.13.1/build/native/Linux-a\ md64-64' [exec] make [1] : *** [all-recursive] Error 1 [exec] make: *** [all] Error 2 BUILD FAILED /home/jssarma/fbprojects/hadoop-0.13.1/build.xml:285: exec returned: 2 at org.apache.tools.ant.taskdefs.ExecTask.runExecute(ExecTask.java:591) at org.apache.tools.ant.taskdefs.ExecTask.runExec(ExecTask.java:617) at org.apache.tools.ant.taskdefs.ExecTask.execute(ExecTask.java:452) at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:275)
        Hide
        jsensarma Joydeep Sen Sarma added a comment -

        sorry - u were probably looking for this:

        checking for lzo/lzo1.h... yes
        checking Checking for the 'actual' dynamic-library for '-llzo2'...
        checking lzo/lzo1a.h usability... yes
        checking lzo/lzo1a.h presence... yes
        checking for lzo/lzo1a.h... yes
        checking Checking for the 'actual' dynamic-library for '-llzo2'... (cached)
        checking lzo/lzo1b.h usability... yes
        checking lzo/lzo1b.h presence... yes
        checking for lzo/lzo1b.h... yes
        checking Checking for the 'actual' dynamic-library for '-llzo2'... (cached)
        checking lzo/lzo1c.h usability... yes
        checking lzo/lzo1c.h presence... yes
        checking for lzo/lzo1c.h... yes
        checking Checking for the 'actual' dynamic-library for '-llzo2'... (cached)
        checking lzo/lzo1f.h usability... yes
        checking lzo/lzo1f.h presence... yes
        checking for lzo/lzo1f.h... yes
        checking Checking for the 'actual' dynamic-library for '-llzo2'... (cached)
        checking lzo/lzo1x.h usability... yes
        checking lzo/lzo1x.h presence... yes
        checking for lzo/lzo1x.h... yes
        checking Checking for the 'actual' dynamic-library for '-llzo2'... (cached)
        checking lzo/lzo1y.h usability... yes
        checking lzo/lzo1y.h presence... yes
        checking for lzo/lzo1y.h... yes
        checking Checking for the 'actual' dynamic-library for '-llzo2'... (cached)
        checking lzo/lzo1z.h usability... yes
        checking lzo/lzo1z.h presence... yes
        checking for lzo/lzo1z.h... yes
        checking Checking for the 'actual' dynamic-library for '-llzo2'... (cached)
        checking lzo/lzo2a.h usability... yes
        checking lzo/lzo2a.h presence... yes
        checking for lzo/lzo2a.h... yes
        checking Checking for the 'actual' dynamic-library for '-llzo2'... (cached)
        checking lzo/lzo_asm.h usability... yes
        checking lzo/lzo_asm.h presence... yes
        checking for lzo/lzo_asm.h... yes
        checking Checking for the 'actual' dynamic-library for '-llzo2'... (cached)
        checking for an ANSI C-conforming const... yes
        checking for memset... yes

        Show
        jsensarma Joydeep Sen Sarma added a comment - sorry - u were probably looking for this: checking for lzo/lzo1.h... yes checking Checking for the 'actual' dynamic-library for '-llzo2'... checking lzo/lzo1a.h usability... yes checking lzo/lzo1a.h presence... yes checking for lzo/lzo1a.h... yes checking Checking for the 'actual' dynamic-library for '-llzo2'... (cached) checking lzo/lzo1b.h usability... yes checking lzo/lzo1b.h presence... yes checking for lzo/lzo1b.h... yes checking Checking for the 'actual' dynamic-library for '-llzo2'... (cached) checking lzo/lzo1c.h usability... yes checking lzo/lzo1c.h presence... yes checking for lzo/lzo1c.h... yes checking Checking for the 'actual' dynamic-library for '-llzo2'... (cached) checking lzo/lzo1f.h usability... yes checking lzo/lzo1f.h presence... yes checking for lzo/lzo1f.h... yes checking Checking for the 'actual' dynamic-library for '-llzo2'... (cached) checking lzo/lzo1x.h usability... yes checking lzo/lzo1x.h presence... yes checking for lzo/lzo1x.h... yes checking Checking for the 'actual' dynamic-library for '-llzo2'... (cached) checking lzo/lzo1y.h usability... yes checking lzo/lzo1y.h presence... yes checking for lzo/lzo1y.h... yes checking Checking for the 'actual' dynamic-library for '-llzo2'... (cached) checking lzo/lzo1z.h usability... yes checking lzo/lzo1z.h presence... yes checking for lzo/lzo1z.h... yes checking Checking for the 'actual' dynamic-library for '-llzo2'... (cached) checking lzo/lzo2a.h usability... yes checking lzo/lzo2a.h presence... yes checking for lzo/lzo2a.h... yes checking Checking for the 'actual' dynamic-library for '-llzo2'... (cached) checking lzo/lzo_asm.h usability... yes checking lzo/lzo_asm.h presence... yes checking for lzo/lzo_asm.h... yes checking Checking for the 'actual' dynamic-library for '-llzo2'... (cached) checking for an ANSI C-conforming const... yes checking for memset... yes
        Hide
        lars_francke Lars Francke added a comment -

        I have a similar problem using FreeBSD 7.1.

        ac_cv_libname_$1="`objdump -p conftest | grep NEEDED | grep $1 | sed 's/\W*NEEDED\W*(.)\W$/\"\1\"/'`"

        A sample objdump output line is:
        NEEDED libc.so.7
        the sed regex resolves this to:
        " libc.so.7"
        with four extra spaces in front of the lib name. This causes the lib to fail loading.
        Manually changing the config.h works but I don't know how to fix this regex.

        To find this out I changed the following line in ZLibCompressor (and the other files):
        THROW(env, "java/lang/UnsatisfiedLinkError", "Cannot load libz.so");
        to
        THROW(env, "java/lang/UnsatisfiedLinkError", dlerror());
        and changed ZlibCompressor.java to include a LOG and actually output the exception message thrown in the native code. As it is now the message is simply discarded and the only message that is shown is "Failed to load/initialize native-zlib library". Not very helpful

        I'm not sure if I should open a new issue for this last part but I believe it would be a sensible default behaviour.

        Show
        lars_francke Lars Francke added a comment - I have a similar problem using FreeBSD 7.1. ac_cv_libname_$1="`objdump -p conftest | grep NEEDED | grep $1 | sed 's/\W*NEEDED\W*(. )\W $/\"\1\"/'`" A sample objdump output line is: NEEDED libc.so.7 the sed regex resolves this to: " libc.so.7" with four extra spaces in front of the lib name. This causes the lib to fail loading. Manually changing the config.h works but I don't know how to fix this regex. To find this out I changed the following line in ZLibCompressor (and the other files): THROW(env, "java/lang/UnsatisfiedLinkError", "Cannot load libz.so"); to THROW(env, "java/lang/UnsatisfiedLinkError", dlerror()); and changed ZlibCompressor.java to include a LOG and actually output the exception message thrown in the native code. As it is now the message is simply discarded and the only message that is shown is "Failed to load/initialize native-zlib library". Not very helpful I'm not sure if I should open a new issue for this last part but I believe it would be a sensible default behaviour.
        Hide
        eli Eli Collins added a comment -

        LZO was removed

        Show
        eli Eli Collins added a comment - LZO was removed
        Hide
        miguno Michael Noll added a comment -

        FYI: The reason might be a problem with ld and how it is configured to work by default. We have run into basically the exact same error when trying to build the hadoop-lzo libraries from [1].

        I described the problem (for hadoop-lzo) in length here:
        https://github.com/kevinweil/hadoop-lzo/issues/33

        In a nutshell, the problem is that some systems run ld with a default setting of "--as-needed" whereas others have "--no-as-needed". The build will only work if you run ld with the latter, i.e. "--no-as-needed". We run into this error when we tried to build hadoop-lzo on Ubuntu 11.10, which is the first Ubuntu version where the default behavior of ld has been changed to --as-needed.

        Now the reason why this has an effect is the way that the Hadoop LZO library is being searched for in src/native/configure. It is using a dummy C script and compiles it by running gcc with "-llzo2":

        // from src/native/configure:
        echo 'int main(int argc, char **argv){return 0;}' > conftest.c
        

        However, the dummy C script does not actually use lzo2 directly, so whether the lzo2 library will be linked or not depends now on whether ld is being run with "--as-needed" or with "--no-as-needed".

        I provided a patch [2] for hadoop-lzo that fixes this problem by setting LDFLAGS (env var) in ant's build.xml. The same fix solves this build error for Hadoop 0.20.203.0, too:

            <exec dir="${build.native}" executable="sh" failonerror="true">
                  <env key="OS_NAME" value="${os.name}"/>
                  <env key="OS_ARCH" value="${os.arch}"/>
                  <env key="JVM_DATA_MODEL" value="${sun.arch.data.model}"/>
                  <env key="HADOOP_NATIVE_SRCDIR" value="${native.src.dir}"/>
                  <env key="LDFLAGS" value="-Wl,--no-as-needed"/>     <== add this line
                  <arg line="${native.src.dir}/configure"/>
            </exec>
        

        If this seems like a reasonable approach (I'm not a C expert) I can provide a similar patch for Hadoop 0.20.20x as well.

        [1] https://github.com/kevinweil/hadoop-lzo
        [2] https://github.com/kevinweil/hadoop-lzo/pull/34

        Show
        miguno Michael Noll added a comment - FYI: The reason might be a problem with ld and how it is configured to work by default. We have run into basically the exact same error when trying to build the hadoop-lzo libraries from [1] . I described the problem (for hadoop-lzo) in length here: https://github.com/kevinweil/hadoop-lzo/issues/33 In a nutshell, the problem is that some systems run ld with a default setting of "--as-needed" whereas others have "--no-as-needed" . The build will only work if you run ld with the latter, i.e. "--no-as-needed" . We run into this error when we tried to build hadoop-lzo on Ubuntu 11.10, which is the first Ubuntu version where the default behavior of ld has been changed to --as-needed . Now the reason why this has an effect is the way that the Hadoop LZO library is being searched for in src/native/configure . It is using a dummy C script and compiles it by running gcc with "-llzo2": // from src/ native /configure: echo ' int main( int argc, char **argv){ return 0;}' > conftest.c However, the dummy C script does not actually use lzo2 directly, so whether the lzo2 library will be linked or not depends now on whether ld is being run with "--as-needed" or with "--no-as-needed" . I provided a patch [2] for hadoop-lzo that fixes this problem by setting LDFLAGS (env var) in ant's build.xml . The same fix solves this build error for Hadoop 0.20.203.0, too: <exec dir= "${build. native }" executable= "sh" failonerror= " true " > <env key= "OS_NAME" value= "${os.name}" /> <env key= "OS_ARCH" value= "${os.arch}" /> <env key= "JVM_DATA_MODEL" value= "${sun.arch.data.model}" /> <env key= "HADOOP_NATIVE_SRCDIR" value= "${ native .src.dir}" /> <env key= "LDFLAGS" value= "-Wl,--no-as-needed" /> <== add this line <arg line= "${ native .src.dir}/configure" /> </exec> If this seems like a reasonable approach (I'm not a C expert) I can provide a similar patch for Hadoop 0.20.20x as well. [1] https://github.com/kevinweil/hadoop-lzo [2] https://github.com/kevinweil/hadoop-lzo/pull/34

          People

          • Assignee:
            acmurthy Arun C Murthy
            Reporter:
            jsensarma Joydeep Sen Sarma
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development