Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.20.0
    • Fix Version/s: 0.20.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Compilation of the native libs is broken:

      compile-core-native:
          [javah] [Search path = /toolshome/build/Linux_2.6_rh4_x86_64/tools/java/jdk1.6.0_i586/jre/lib/resources.jar: \
                                 /toolshome/build/Linux_2.6_rh4_x86_64/tools/java/jdk1.6.0_i586/jre/lib/rt.jar: \
                                 /toolshome/build/Linux_2.6_rh4_x86_64/tools/java/jdk1.6.0_i586/jre/lib/sunrsasign.jar: \
                                 /toolshome/build/Linux_2.6_rh4_x86_64/tools/java/jdk1.6.0_i586/jre/lib/jsse.jar: \
                                 /toolshome/build/Linux_2.6_rh4_x86_64/tools/java/jdk1.6.0_i586/jre/lib/jce.jar: \
                                 /toolshome/build/Linux_2.6_rh4_x86_64/tools/java/jdk1.6.0_i586/jre/lib/charsets.jar: \
                                 /toolshome/build/Linux_2.6_rh4_x86_64/tools/java/jdk1.6.0_i586/jre/classes: \
                                 /hadoophome/build/classes]
          [javah] [Loaded /hadoophome/build/classes/org/apache/hadoop/io/compress/zlib/ZlibCompressor.class]
          [javah] [Loaded /toolshome/build/Linux_2.6_rh4_x86_64/tools/java/jdk1.6.0_i586/jre/lib/rt.jar(java/lang/Object.class)]
          [javah] [Forcefully writing file /hadoophome/build/native/Linux-i386-32/src/org/apache/hadoop/io/compress/zlib/org_apache_hadoop_io_compress_zlib_ZlibCompressor.h]
          [javah] [Loaded /hadoophome/build/classes/org/apache/hadoop/io/compress/zlib/ZlibDecompressor.class]
          [javah] [Forcefully writing file /hadoophome/build/native/Linux-i386-32/src/org/apache/hadoop/io/compress/zlib/org_apache_hadoop_io_compress_zlib_ZlibDecompressor.h]
          [javah] [Search path = /toolshome/build/Linux_2.6_rh4_x86_64/tools/java/jdk1.6.0_i586/jre/lib/resources.jar: \
                                 /toolshome/build/Linux_2.6_rh4_x86_64/tools/java/jdk1.6.0_i586/jre/lib/rt.jar: \
                                 /toolshome/build/Linux_2.6_rh4_x86_64/tools/java/jdk1.6.0_i586/jre/lib/sunrsasign.jar: \
                                 /toolshome/build/Linux_2.6_rh4_x86_64/tools/java/jdk1.6.0_i586/jre/lib/jsse.jar: \
                                 /toolshome/build/Linux_2.6_rh4_x86_64/tools/java/jdk1.6.0_i586/jre/lib/jce.jar: \
                                 /toolshome/build/Linux_2.6_rh4_x86_64/tools/java/jdk1.6.0_i586/jre/lib/charsets.jar: \
                                 /toolshome/build/Linux_2.6_rh4_x86_64/tools/java/jdk1.6.0_i586/jre/classes: \
                                 /hadoophome/build/classes]
          [javah] Error: Class org.apache.hadoop.io.compress.lzo.LzoCompressor could not be found.
      
      1. 4949_20081230.patch
        1 kB
        Tsz Wo Nicholas Sze
      2. 4949_20081231.patch
        90 kB
        Tsz Wo Nicholas Sze
      3. 4949_buildxml.patch
        2 kB
        Lee Tucker
      4. 4949-0.patch
        0.9 kB
        Chris Douglas
      5. 4949-1.patch
        254 kB
        Chris Douglas
      6. HADOOP-4949.patch
        760 kB
        Giridharan Kesavan

        Issue Links

          Activity

          Hide
          Chris Douglas added a comment -

          The trivial fix (attached) doesn't resolve this for me:

               [exec] creating libnativezlib.la
               [exec] (cd .libs && rm -f libnativezlib.la && ln -s ../libnativezlib.la libnativezlib.la)
               [exec] make[2]: Leaving directory `/snip/hadoop/build/native/Linux-i386-32/src/org/apache/hadoop/io/compress/zlib'
               [exec] make[2]: Entering directory `/snip/hadoop/build/native/Linux-i386-32'
               [exec] make[2]: Leaving directory `/snip/hadoop/build/native/Linux-i386-32'
               [exec] make[1]: Leaving directory `/snip/hadoop/build/native/Linux-i386-32'
               [exec] libtool: install: `/snip/hadoop/build/native/Linux-i386-32/lib/libhadoop.la' is not a valid libtool archive
               [exec] Try `libtool --help --mode=install' for more information.
          
          BUILD FAILED
          /snip/hadoop/build.xml:484: exec returned: 1
          
          Show
          Chris Douglas added a comment - The trivial fix (attached) doesn't resolve this for me: [exec] creating libnativezlib.la [exec] (cd .libs && rm -f libnativezlib.la && ln -s ../libnativezlib.la libnativezlib.la) [exec] make[2]: Leaving directory `/snip/hadoop/build/native/Linux-i386-32/src/org/apache/hadoop/io/compress/zlib' [exec] make[2]: Entering directory `/snip/hadoop/build/native/Linux-i386-32' [exec] make[2]: Leaving directory `/snip/hadoop/build/native/Linux-i386-32' [exec] make[1]: Leaving directory `/snip/hadoop/build/native/Linux-i386-32' [exec] libtool: install: `/snip/hadoop/build/native/Linux-i386-32/lib/libhadoop.la' is not a valid libtool archive [exec] Try `libtool --help --mode=install' for more information. BUILD FAILED /snip/hadoop/build.xml:484: exec returned: 1
          Hide
          Chris Douglas added a comment -

          [edit] Cleaned up description

          Show
          Chris Douglas added a comment - [edit] Cleaned up description
          Hide
          Giridharan Kesavan added a comment -

          This patch fixes the error compiling compile-core-native target. This is an improved version over the patch submitted by Chris, thanks to Chris

          local test-patch O/P

          [exec] +1 overall.
          [exec]
          [exec] +1 @author. The patch does not contain any @author tags.
          [exec]
          [exec] +1 tests included. The patch appears to include 1 new or modified tests.
          [exec]
          [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
          [exec]
          [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
          [exec]
          [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
          [exec]
          [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
          [exec]

          Thanks,
          Giri

          Show
          Giridharan Kesavan added a comment - This patch fixes the error compiling compile-core-native target. This is an improved version over the patch submitted by Chris, thanks to Chris local test-patch O/P [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 1 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity. [exec] Thanks, Giri
          Hide
          Lee Tucker added a comment -

          Tested Giri's patch and was able to successfully build native hdfs.

          Show
          Lee Tucker added a comment - Tested Giri's patch and was able to successfully build native hdfs.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Tried the patches. Both patches failed on my machine. It might be the case that my machine is not set up correctly for compiling the native library.

          The following is the results from Giri's patch.

          $ ant compile-native
          Buildfile: build.xml
          
          compile-native:
          
          ...
          
               [exec] config.status: config.h is unchanged
               [exec] config.status: executing depfiles commands
               [exec]  cd /home/tsz/hadoop/latest/src/native && /bin/sh /home/tsz/hadoop/latest/src/native/config/missing --run automake-1.9 --gnu 
               [exec] Usage: autoconf [-h] [--help] [-m dir] [--macrodir=dir]
               [exec]        [-l dir] [--localdir=dir] [--version] [template-file]
               [exec] automake-1.9: autoconf failed with exit status: 1
               [exec] make: *** [/home/tsz/hadoop/latest/src/native/Makefile.in] Error 1
          
          BUILD FAILED
          /home/tsz/hadoop/latest/build.xml:448: The following error occurred while executing this line:
          /home/tsz/hadoop/latest/build.xml:477: exec returned: 2
          
          Total time: 21 seconds
          

          I got the same problem for Chris' patch.

          Show
          Tsz Wo Nicholas Sze added a comment - Tried the patches. Both patches failed on my machine. It might be the case that my machine is not set up correctly for compiling the native library. The following is the results from Giri's patch. $ ant compile-native Buildfile: build.xml compile-native: ... [exec] config.status: config.h is unchanged [exec] config.status: executing depfiles commands [exec] cd /home/tsz/hadoop/latest/src/native && /bin/sh /home/tsz/hadoop/latest/src/native/config/missing --run automake-1.9 --gnu [exec] Usage: autoconf [-h] [--help] [-m dir] [--macrodir=dir] [exec] [-l dir] [--localdir=dir] [--version] [template-file] [exec] automake-1.9: autoconf failed with exit status: 1 [exec] make: *** [/home/tsz/hadoop/latest/src/native/Makefile.in] Error 1 BUILD FAILED /home/tsz/hadoop/latest/build.xml:448: The following error occurred while executing this line: /home/tsz/hadoop/latest/build.xml:477: exec returned: 2 Total time: 21 seconds I got the same problem for Chris' patch.
          Hide
          Lee Tucker added a comment -

          Nicholas, it looks like you don't have the right version of autconf. Hadoop-3344 requires updates to autoconf and ant, and has been applied to trunk.

          Show
          Lee Tucker added a comment - Nicholas, it looks like you don't have the right version of autconf. Hadoop-3344 requires updates to autoconf and ant, and has been applied to trunk.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          4949_20081230.patch: For easier review, I removed all the generated codes from Giri's patch.

          Show
          Tsz Wo Nicholas Sze added a comment - 4949_20081230.patch: For easier review, I removed all the generated codes from Giri's patch.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1

          I updated autoconf in my machine. With Giri's patch (or 4949_20081230.patch), both the followings succeed in my machine.

          • ant compile -Dcompile.c++=true -Dlibhdfs=true
          • ant compile-native
          Show
          Tsz Wo Nicholas Sze added a comment - +1 I updated autoconf in my machine. With Giri's patch (or 4949_20081230.patch), both the followings succeed in my machine. ant compile -Dcompile.c++=true -Dlibhdfs=true ant compile-native
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Checked again 4949_20081230.patch only works for "ant compile -Dcompile.c++=true -Dlibhdfs=true" but not "ant compile-native". Giri's patch works for both.

          Show
          Tsz Wo Nicholas Sze added a comment - Checked again 4949_20081230.patch only works for "ant compile -Dcompile.c++=true -Dlibhdfs=true" but not "ant compile-native". Giri's patch works for both.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          4949_20081231.patch: added the change in src/native/aclocal.m4. This patch works for both ant commands.

          Show
          Tsz Wo Nicholas Sze added a comment - 4949_20081231.patch: added the change in src/native/aclocal.m4. This patch works for both ant commands.
          Hide
          Raghu Angadi added a comment -

          I just committed this. Thanks Chris, Giri, and Nicholas.

          Show
          Raghu Angadi added a comment - I just committed this. Thanks Chris, Giri, and Nicholas.
          Hide
          Tsz Wo Nicholas Sze added a comment -
          • After compiled the native library, some generated files are overwritten. For examples,
            M      src/native/configure
            M      src/native/Makefile.in
            M      src/native/src/org/apache/hadoop/io/compress/zlib/Makefile.in
            M      src/native/lib/Makefile.in
            M      src/c++/pipes/aclocal.m4
            M      src/c++/pipes/configure
            M      src/c++/utils/configure
            ?      src/c++/libhdfs/autom4te.cache
            M      src/c++/libhdfs/Makefile.in
            

            The changes depend on which the ant command parameters used. Should we add these files to the ignore list?

          • The compilation depends on the ordering of executions. In my machine,
            • it works if
              1. ant compile -Dcompile.c++=true -Dlibhdfs=true
              2. ant compile-native
            • but if fails if
              1. ant compile-native
              2. ant compile -Dcompile.c++=true -Dlibhdfs=true
          • There are different ways to compile native library. It is not clear to me that which one should be used. e.g.
            • ant -Dcompile.c++=yes -Dcompile.native=yes -Dlibhdfs=yes compile
            • ant compile-native -Dcompile.c++=true -Dlibhdfs=true
            • ant compile -Dcompile.c++=true -Dlibhdfs=true
            • ant compile-native
          Show
          Tsz Wo Nicholas Sze added a comment - After compiled the native library, some generated files are overwritten. For examples, M src/native/configure M src/native/Makefile.in M src/native/src/org/apache/hadoop/io/compress/zlib/Makefile.in M src/native/lib/Makefile.in M src/c++/pipes/aclocal.m4 M src/c++/pipes/configure M src/c++/utils/configure ? src/c++/libhdfs/autom4te.cache M src/c++/libhdfs/Makefile.in The changes depend on which the ant command parameters used. Should we add these files to the ignore list? The compilation depends on the ordering of executions. In my machine, it works if ant compile -Dcompile.c++=true -Dlibhdfs=true ant compile-native but if fails if ant compile-native ant compile -Dcompile.c++=true -Dlibhdfs=true There are different ways to compile native library. It is not clear to me that which one should be used. e.g. ant -Dcompile.c++=yes -Dcompile.native=yes -Dlibhdfs=yes compile ant compile-native -Dcompile.c++=true -Dlibhdfs=true ant compile -Dcompile.c++=true -Dlibhdfs=true ant compile-native
          Hide
          Lee Tucker added a comment -

          I'm sorry, but none of the methods above work for me.

          If I try "ant compile-native" on the trunk I get the following failure on the compile-core-native target" regardless of which of the above methods I use to invoke it. It also fails if I invoke: "ant -Dcompile.c++=yes -Dcompile.native=yes -Dlibhdfs=yes compile"

          [exec] libtool: install: `/grid/0/gs/gridre/Spring/hadoopSource/build/native/Linux-i386-32/lib/libhadoop.la' is not a valid libtool archive
          [exec] Try `libtool --help --mode=install' for more information.

          The only command that seems to work correctly is: "ant compile -Dcompile.c++=true -Dlibhdfs=true". I suspect that the compile-native target is the root cause, but will need to investigate further.

          Show
          Lee Tucker added a comment - I'm sorry, but none of the methods above work for me. If I try "ant compile-native" on the trunk I get the following failure on the compile-core-native target" regardless of which of the above methods I use to invoke it. It also fails if I invoke: "ant -Dcompile.c++=yes -Dcompile.native=yes -Dlibhdfs=yes compile" [exec] libtool: install: `/grid/0/gs/gridre/Spring/hadoopSource/build/native/Linux-i386-32/lib/libhadoop.la' is not a valid libtool archive [exec] Try `libtool --help --mode=install' for more information. The only command that seems to work correctly is: "ant compile -Dcompile.c++=true -Dlibhdfs=true". I suspect that the compile-native target is the root cause, but will need to investigate further.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          The following should work for current trunk:

          1. checkout a fresh copy or revert all local changes (you need this as long as you ever have run ant to compile native library since some scripts may be incorrectly generated.)
          2. ant compile -Dcompile.c++=true -Dlibhdfs=true
          3. ant compile-native
            After the above, any ant commands will work.

          I believe the problem is in build.xml. It does not generate the scripts correctly.

          Show
          Tsz Wo Nicholas Sze added a comment - The following should work for current trunk: checkout a fresh copy or revert all local changes (you need this as long as you ever have run ant to compile native library since some scripts may be incorrectly generated.) ant compile -Dcompile.c++=true -Dlibhdfs=true ant compile-native After the above, any ant commands will work. I believe the problem is in build.xml. It does not generate the scripts correctly.
          Hide
          Lee Tucker added a comment -

          Giri, I'm reopening this because 4949 isn't completely fixed.

          Show
          Lee Tucker added a comment - Giri, I'm reopening this because 4949 isn't completely fixed.
          Hide
          Chris Douglas added a comment -

          The cases Nicholas cited earlier, particularly the ordering problem, are regressions from r727255 and should be fixed. Since the current fix touches rarely modified code, I'd like to keep it in trunk until this is resolved.

          Show
          Chris Douglas added a comment - The cases Nicholas cited earlier, particularly the ordering problem, are regressions from r727255 and should be fixed. Since the current fix touches rarely modified code, I'd like to keep it in trunk until this is resolved.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk #708 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/708/)
          . Fix native compilation. (Chris Douglas via rangadi)

          Show
          Hudson added a comment - Integrated in Hadoop-trunk #708 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/708/ ) . Fix native compilation. (Chris Douglas via rangadi)
          Hide
          Lee Tucker added a comment -

          removing all traces of previous bad patch.

          Show
          Lee Tucker added a comment - removing all traces of previous bad patch.
          Hide
          Lee Tucker added a comment - - edited

          This patch changes the clean target to clean up the makefiles by running autoreconf -if. If I understand correctly, this just resets the build artifacts to match the gnutools installed. The compile-native target now also does and autoreconf, just like the c++ targets do.

          If you do either of the following commands, the build now works correctly.

          ant -Dcompile.c++=true -Dcompile.native=yes -Dlibhdfs=true clean compile-native compile
          ant -Dcompile.c++=true -Dcompile.native=yes -Dlibhdfs=true clean compile compile-native

          (My assumption is that non-c++ and non-native targets aren't impacted by this change.)

          It needs to be applied in addition to the previously committed patch. This patch also works, with some "fuzzing" on the 0.20 branch.

          Show
          Lee Tucker added a comment - - edited This patch changes the clean target to clean up the makefiles by running autoreconf -if. If I understand correctly, this just resets the build artifacts to match the gnutools installed. The compile-native target now also does and autoreconf, just like the c++ targets do. If you do either of the following commands, the build now works correctly. ant -Dcompile.c++=true -Dcompile.native=yes -Dlibhdfs=true clean compile-native compile ant -Dcompile.c++=true -Dcompile.native=yes -Dlibhdfs=true clean compile compile-native (My assumption is that non-c++ and non-native targets aren't impacted by this change.) It needs to be applied in addition to the previously committed patch. This patch also works, with some "fuzzing" on the 0.20 branch.
          Hide
          Chris Douglas added a comment -

          Attached patch assumes that the currently committed patch has been reverted; it works for me, but further testing may be required.

          Show
          Chris Douglas added a comment - Attached patch assumes that the currently committed patch has been reverted; it works for me, but further testing may be required.
          Hide
          Arun C Murthy added a comment -

          I just committed this. Thanks, Chris!

          Show
          Arun C Murthy added a comment - I just committed this. Thanks, Chris!

            People

            • Assignee:
              Chris Douglas
              Reporter:
              Chris Douglas
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development