Joydeep,

I don't have access to an amd64-fedora box, could you please try this and let me know?

$ echo 'int main(int argc, char **argv){return 0;}' > conftest.c
$ gcc -o conftest conftest.c -llzo2
$ objdump -p conftest | grep NEEDED

on my system I get:

$ objdump -p conftest | grep NEEDED
  NEEDED      liblzo2.so.2
  NEEDED      libc.so.6

as i reported - i have already tried this out (retried just in case). On my platform - i only see libc.so.6 as the required library.

I am wondering if it is because i don't have .so versions of lz libraries - only .a:
[jssarma@dev046 if]$ ls -l /usr/local/lib/lzo2
rw-rr- 1 root root 847206 Feb 28 2007 /usr/local/lib/liblzo2.a
-rwxr-xr-x 1 root root 752 Feb 28 2007 /usr/local/lib/liblzo2.la

Yes, that is weird. The rpm which got you lzo2 shud come with both dynamic and static versions of lzo... could you check and maybe reinstall lzo? Thanks.

As I mentioned - i already have a workaround - i edited configure to hard code the library path.

I don't know how lzo was installed - the system was imaged by IT folks and it is the way it is. The issue is how to help the next person who hits this error. Clearly, if grep NEEDED returns empty - there's an error - and either the script/build needs to stop. Perhaps you might want to just print out an error from configure and ask them to change the script. or get it off a environment variable.

The idea of using gnu's auto* toolset is to not rely on hard-coded paths since they are system/distribution specific.

So, well, any information you provide does help in solving the problem for the next user. Clearly it is easier to fix if you could confirm that the compile went through with liblzo2.so being present on your system, else there might be other doors to knock-on.

Btw, could you also attach the output of the entire compilation process i.e. the parts where configure checks for zlib/lzo headers and so on?

here's the log. i think that the error should be detected in configure and auto-configuration should either fail outright or print a loud warning that compilation of native libs will not work unless some steps are taken. It might take me some more time to test with .so

[exec] if /bin/sh ../../../../../../../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I.\
-I/home/jssarma/fbprojects/hadoop-0.13.1/src/native/src/org/apache/hadoop/io/compress/lzo -I../../\
../../../../.. -I/usr/local/java/include -I/usr/local/java/include/linux -I/home/jssarma/fbproject\
s/hadoop-0.13.1/src/native/src -g -Wall -fPIC -O2 -m64 -g -O2 -MT LzoCompressor.lo -MD -MP -MF ".d\
eps/LzoCompressor.Tpo" -c -o LzoCompressor.lo /home/jssarma/fbprojects/hadoop-0.13.1/src/native/src\
/org/apache/hadoop/io/compress/lzo/LzoCompressor.c; \
[exec] then mv -f ".deps/LzoCompressor.Tpo" ".deps/LzoCompressor.Plo"; else rm -f ".deps/LzoCo\
mpressor.Tpo"; exit 1; fi
[exec] gcc -DHAVE_CONFIG_H -I. -I/home/jssarma/fbprojects/hadoop-0.13.1/src/native/src/org/ap\
ache/hadoop/io/compress/lzo -I../../../../../../.. -I/usr/local/java/include -I/usr/local/java/incl\
ude/linux -I/home/jssarma/fbprojects/hadoop-0.13.1/src/native/src -g -Wall -fPIC -O2 -m64 -g -O2 -M\
T LzoCompressor.lo -MD -MP -MF .deps/LzoCompressor.Tpo -c /home/jssarma/fbprojects/hadoop-0.13.1/sr\
c/native/src/org/apache/hadoop/io/compress/lzo/LzoCompressor.c -fPIC -DPIC -o .libs/LzoCompressor.\
o
[exec] /home/jssarma/fbprojects/hadoop-0.13.1/src/native/src/org/apache/hadoop/io/compress/lzo\
/LzoCompressor.c: In function 'Java_org_apache_hadoop_io_compress_lzo_LzoCompressor_initIDs':
[exec] /home/jssarma/fbprojects/hadoop-0.13.1/src/native/src/org/apache/hadoop/io/compress/lzo\
/LzoCompressor.c:116: error: syntax error before '/' token
[exec] make[2]: Leaving directory `/home/jssarma/fbprojects/hadoop-0.13.1/build/native/Linux-a\
md64-64/src/org/apache/hadoop/io/compress/lzo'
[exec] make[2]: *** [LzoCompressor.lo] Error 1
[exec] make[1]: Leaving directory `/home/jssarma/fbprojects/hadoop-0.13.1/build/native/Linux-a\
md64-64'
[exec] make[1]: *** [all-recursive] Error 1
[exec] make: *** [all] Error 2

BUILD FAILED
/home/jssarma/fbprojects/hadoop-0.13.1/build.xml:285: exec returned: 2
at org.apache.tools.ant.taskdefs.ExecTask.runExecute(ExecTask.java:591)
at org.apache.tools.ant.taskdefs.ExecTask.runExec(ExecTask.java:617)
at org.apache.tools.ant.taskdefs.ExecTask.execute(ExecTask.java:452)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:275)

sorry - u were probably looking for this:

checking for lzo/lzo1.h... yes
checking Checking for the 'actual' dynamic-library for '-llzo2'...
checking lzo/lzo1a.h usability... yes
checking lzo/lzo1a.h presence... yes
checking for lzo/lzo1a.h... yes
checking Checking for the 'actual' dynamic-library for '-llzo2'... (cached)
checking lzo/lzo1b.h usability... yes
checking lzo/lzo1b.h presence... yes
checking for lzo/lzo1b.h... yes
checking Checking for the 'actual' dynamic-library for '-llzo2'... (cached)
checking lzo/lzo1c.h usability... yes
checking lzo/lzo1c.h presence... yes
checking for lzo/lzo1c.h... yes
checking Checking for the 'actual' dynamic-library for '-llzo2'... (cached)
checking lzo/lzo1f.h usability... yes
checking lzo/lzo1f.h presence... yes
checking for lzo/lzo1f.h... yes
checking Checking for the 'actual' dynamic-library for '-llzo2'... (cached)
checking lzo/lzo1x.h usability... yes
checking lzo/lzo1x.h presence... yes
checking for lzo/lzo1x.h... yes
checking Checking for the 'actual' dynamic-library for '-llzo2'... (cached)
checking lzo/lzo1y.h usability... yes
checking lzo/lzo1y.h presence... yes
checking for lzo/lzo1y.h... yes
checking Checking for the 'actual' dynamic-library for '-llzo2'... (cached)
checking lzo/lzo1z.h usability... yes
checking lzo/lzo1z.h presence... yes
checking for lzo/lzo1z.h... yes
checking Checking for the 'actual' dynamic-library for '-llzo2'... (cached)
checking lzo/lzo2a.h usability... yes
checking lzo/lzo2a.h presence... yes
checking for lzo/lzo2a.h... yes
checking Checking for the 'actual' dynamic-library for '-llzo2'... (cached)
checking lzo/lzo_asm.h usability... yes
checking lzo/lzo_asm.h presence... yes
checking for lzo/lzo_asm.h... yes
checking Checking for the 'actual' dynamic-library for '-llzo2'... (cached)
checking for an ANSI C-conforming const... yes
checking for memset... yes

I have a similar problem using FreeBSD 7.1.

ac_cv_libname_$1="`objdump -p conftest | grep NEEDED | grep $1 | sed 's/\W*NEEDED\W*(.)\W$/\"\1\"/'`"

A sample objdump output line is:
NEEDED libc.so.7
the sed regex resolves this to:
" libc.so.7"
with four extra spaces in front of the lib name. This causes the lib to fail loading.
Manually changing the config.h works but I don't know how to fix this regex.

To find this out I changed the following line in ZLibCompressor (and the other files):
THROW(env, "java/lang/UnsatisfiedLinkError", "Cannot load libz.so");
to
THROW(env, "java/lang/UnsatisfiedLinkError", dlerror());
and changed ZlibCompressor.java to include a LOG and actually output the exception message thrown in the native code. As it is now the message is simply discarded and the only message that is shown is "Failed to load/initialize native-zlib library". Not very helpful

I'm not sure if I should open a new issue for this last part but I believe it would be a sensible default behaviour.

LZO was removed

FYI: The reason might be a problem with ld and how it is configured to work by default. We have run into basically the exact same error when trying to build the hadoop-lzo libraries from [1].

I described the problem (for hadoop-lzo) in length here:
https://github.com/kevinweil/hadoop-lzo/issues/33

In a nutshell, the problem is that some systems run ld with a default setting of "--as-needed" whereas others have "--no-as-needed". The build will only work if you run ld with the latter, i.e. "--no-as-needed". We run into this error when we tried to build hadoop-lzo on Ubuntu 11.10, which is the first Ubuntu version where the default behavior of ld has been changed to --as-needed.

Now the reason why this has an effect is the way that the Hadoop LZO library is being searched for in src/native/configure. It is using a dummy C script and compiles it by running gcc with "-llzo2":

// from src/native/configure:
echo 'int main(int argc, char **argv){return 0;}' > conftest.c

However, the dummy C script does not actually use lzo2 directly, so whether the lzo2 library will be linked or not depends now on whether ld is being run with "--as-needed" or with "--no-as-needed".

I provided a patch [2] for hadoop-lzo that fixes this problem by setting LDFLAGS (env var) in ant's build.xml. The same fix solves this build error for Hadoop 0.20.203.0, too:

    <exec dir="${build.native}" executable="sh" failonerror="true">
          <env key="OS_NAME" value="${os.name}"/>
          <env key="OS_ARCH" value="${os.arch}"/>
          <env key="JVM_DATA_MODEL" value="${sun.arch.data.model}"/>
          <env key="HADOOP_NATIVE_SRCDIR" value="${native.src.dir}"/>
          <env key="LDFLAGS" value="-Wl,--no-as-needed"/>     <== add this line
          <arg line="${native.src.dir}/configure"/>
    </exec>

If this seems like a reasonable approach (I'm not a C expert) I can provide a similar patch for Hadoop 0.20.20x as well.

[1] https://github.com/kevinweil/hadoop-lzo
[2] https://github.com/kevinweil/hadoop-lzo/pull/34