Hadoop Common
  1. Hadoop Common
  2. HADOOP-3659

Patch to allow hadoop native to compile on Mac OS X

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.20.0
    • Fix Version/s: 0.21.0
    • Component/s: native
    • Labels:
      None
    • Environment:

      Mac OS X 10.5.3

    • Hadoop Flags:
      Reviewed

      Description

      This patch makes the autoconf script work on Mac OS X. LZO needs to be installed (including the optional shared libraries) for the compile to succeed. You'll want to regenerate the configure script using autoconf after applying this patch.

      1. hadoop-native-mac.patch
        0.9 kB
        Colin Evans
      2. hadoop-native-mac.patch
        0.9 kB
        Allen Wittenauer
      3. HADOOP-3659.patch
        0.9 kB
        Allen Wittenauer
      4. hadoop-0.20-compile-native-on-mac-no-autogen-files.patch
        2 kB
        Hong Tang
      5. hadoop-0.20-compile-native-on-mac.patch
        852 kB
        Hong Tang

        Activity

        Tom White made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Tom White made changes -
        Fix Version/s 0.21.0 [ 12313563 ]
        Fix Version/s 0.22.0 [ 12314296 ]
        Hide
        Hong Tang added a comment -

        I see. I am using Leopard and it looks like that it does not support 32-bit 1.6.

        % file Versions/1.6.0/Libraries/libjvm.dylib 
        Versions/1.6.0/Libraries/libjvm.dylib: symbolic link to libclient.dylib
        % file Versions/1.6.0/Libraries/libclient.dylib 
        Versions/1.6.0/Libraries/libclient.dylib: Mach-O 64-bit dynamically linked shared library x86_64
        
        Show
        Hong Tang added a comment - I see. I am using Leopard and it looks like that it does not support 32-bit 1.6. % file Versions/1.6.0/Libraries/libjvm.dylib Versions/1.6.0/Libraries/libjvm.dylib: symbolic link to libclient.dylib % file Versions/1.6.0/Libraries/libclient.dylib Versions/1.6.0/Libraries/libclient.dylib: Mach-O 64-bit dynamically linked shared library x86_64
        Hide
        Allen Wittenauer added a comment -

        Java 1.6 on Snow Leopard supports both 32-bit and 64-bit. (i.e., like Solaris, it honors -d from the same executable) If it doesn't, Apple sure went through a lot of work:

        $ file libjvm.dylib
        libjvm.dylib: Mach-O universal binary with 2 architectures
        libjvm.dylib (for architecture i386): Mach-O dynamically linked shared library i386
        libjvm.dylib (for architecture x86_64): Mach-O 64-bit dynamically linked shared library x86_64

        Show
        Allen Wittenauer added a comment - Java 1.6 on Snow Leopard supports both 32-bit and 64-bit. (i.e., like Solaris, it honors -d from the same executable) If it doesn't, Apple sure went through a lot of work: $ file libjvm.dylib libjvm.dylib: Mach-O universal binary with 2 architectures libjvm.dylib (for architecture i386): Mach-O dynamically linked shared library i386 libjvm.dylib (for architecture x86_64): Mach-O 64-bit dynamically linked shared library x86_64
        Hong Tang made changes -
        Hide
        Hong Tang added a comment -

        BTW, in case it might be helpful, attached are two patches I have that allows Hadoop 0.20 to be compilable with -Dcompile.native=true. There are two variations, one with the bare minimal changes, and you need to run aclocal, autoconf and automake, and the other includes the artifacts of these commands.

        Show
        Hong Tang added a comment - BTW, in case it might be helpful, attached are two patches I have that allows Hadoop 0.20 to be compilable with -Dcompile.native=true. There are two variations, one with the bare minimal changes, and you need to run aclocal, autoconf and automake, and the other includes the artifacts of these commands.
        Hide
        Hong Tang added a comment -

        Hadoop requires Java 1.6, and on Mac OS X, Java 1.6 only comes with 64-bit version. So there is really no way to get 32-bit hadoop with native compilation, right?

        Show
        Hong Tang added a comment - Hadoop requires Java 1.6, and on Mac OS X, Java 1.6 only comes with 64-bit version. So there is really no way to get 32-bit hadoop with native compilation, right?
        Hide
        Allen Wittenauer added a comment -

        A couple of people have had problems getting a 32-bit or 64-bit build. I'd recommend setting ANT_OPTS=-d32 or ANT_OPTS=-d64. This seems to help the ant native compilation targets to build the proper version.

        Show
        Allen Wittenauer added a comment - A couple of people have had problems getting a 32-bit or 64-bit build. I'd recommend setting ANT_OPTS=-d32 or ANT_OPTS=-d64. This seems to help the ant native compilation targets to build the proper version.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk #282 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/282/)
        . Patch to allow hadoop native to compile on Mac OS X. Contributed by Colin Evans and Allen Wittenauer.

        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk #282 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/282/ ) . Patch to allow hadoop native to compile on Mac OS X. Contributed by Colin Evans and Allen Wittenauer.
        Hide
        Allen Wittenauer added a comment -

        At some point, I'll have a future patch which eliminates the LDFLAGS part. I've sort of got it working here, but not quite ready for prime time.

        Thanks Tom.

        Show
        Allen Wittenauer added a comment - At some point, I'll have a future patch which eliminates the LDFLAGS part. I've sort of got it working here, but not quite ready for prime time. Thanks Tom.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #202 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/202/)
        . Patch to allow hadoop native to compile on Mac OS X. Contributed by Colin Evans and Allen Wittenauer.

        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #202 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/202/ ) . Patch to allow hadoop native to compile on Mac OS X. Contributed by Colin Evans and Allen Wittenauer.
        Tom White made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Fix Version/s 0.22.0 [ 12314296 ]
        Resolution Fixed [ 1 ]
        Hide
        Tom White added a comment -

        I've just committed this. Thanks Colin and Allen!

        Show
        Tom White added a comment - I've just committed this. Thanks Colin and Allen!
        Hide
        Eli Collins added a comment -

        I confirmed that HADOOP-3659.patch does not break the 64-bit linux native build and libhdfs test still passes.

        Show
        Eli Collins added a comment - I confirmed that HADOOP-3659 .patch does not break the 64-bit linux native build and libhdfs test still passes.
        Chris Douglas made changes -
        Fix Version/s 0.20.2 [ 12314203 ]
        Hide
        Tom White added a comment -

        I can confirm that this patch works for me on Mac OS X 10.5.8 if I do:

        export LDFLAGS=-L/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Libraries
        ant compile-native
        
        Show
        Tom White added a comment - I can confirm that this patch works for me on Mac OS X 10.5.8 if I do: export LDFLAGS=-L/ System /Library/Frameworks/JavaVM.framework/Versions/1.6.0/Libraries ant compile- native
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12423498/HADOOP-3659.patch
        against trunk revision 829289.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/109/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/109/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/109/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/109/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12423498/HADOOP-3659.patch against trunk revision 829289. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/109/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/109/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/109/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/109/console This message is automatically generated.
        Allen Wittenauer made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Affects Version/s 0.20.0 [ 12313438 ]
        Affects Version/s 0.17.0 [ 12312913 ]
        Fix Version/s 0.20.2 [ 12314203 ]
        Hide
        Allen Wittenauer added a comment -

        I don't think test-patch runs the necessary automagic. Someone should configure this works.

        Show
        Allen Wittenauer added a comment - I don't think test-patch runs the necessary automagic. Someone should configure this works.
        Allen Wittenauer made changes -
        Attachment HADOOP-3659.patch [ 12423498 ]
        Hide
        Allen Wittenauer added a comment -

        I figured out the issue with the code i actually use (which is not the patch)

        But the lzo code example is actually better. I'll build a new patch here in a bit.

        Show
        Allen Wittenauer added a comment - I figured out the issue with the code i actually use (which is not the patch) But the lzo code example is actually better. I'll build a new patch here in a bit.
        Hide
        Hong Tang added a comment -

        @Allen, the lzo codec compiles without glitch on mac os-x. You may want to take a look there. The following might be relevant (in file src/native/m4/compression_utils.m4):

            if test ! -z "`which otool | grep -v 'no otool'`"; then
              ac_cv_libname_$1=\"`otool -L conftest | grep $1 | sed -e 's/^[     ]*//' -e 's/ .*//'`\";
            elif test ! -z "`which objdump | grep -v 'no objdump'`"; then
              ac_cv_libname_$1="`objdump -p conftest | grep NEEDED | grep $1 | sed 's/\W*NEEDED\W*\(.*\)\W*$/\"\1\"/'`"
            elif test ! -z "`which ldd | grep -v 'no ldd'`"; then
              ac_cv_libname_$1="`ldd conftest | grep $1 | sed 's/^[[[^A-Za-z0-9]]]*\([[[A-Za-z0-9\.]]]*\)[[[^A-Za-z0-9]]]*=>.*$/\"\1\"/'`"
            else
              AC_MSG_ERROR(Can't find either 'objdump' or 'ldd' to compute the dynamic library for '-l$1')
            fi
         
        Show
        Hong Tang added a comment - @Allen, the lzo codec compiles without glitch on mac os-x. You may want to take a look there. The following might be relevant (in file src/native/m4/compression_utils.m4): if test ! -z "`which otool | grep -v 'no otool'`"; then ac_cv_libname_$1=\"`otool -L conftest | grep $1 | sed -e 's/^[ ]*//' -e 's/ .*//'`\"; elif test ! -z "`which objdump | grep -v 'no objdump'`"; then ac_cv_libname_$1="`objdump -p conftest | grep NEEDED | grep $1 | sed 's/\W*NEEDED\W*\(.*\)\W*$/\"\1\"/'`" elif test ! -z "`which ldd | grep -v 'no ldd'`"; then ac_cv_libname_$1="`ldd conftest | grep $1 | sed 's/^[[[^A-Za-z0-9]]]*\([[[A-Za-z0-9\.]]]*\)[[[^A-Za-z0-9]]]*=>.*$/\"\1\"/'`" else AC_MSG_ERROR(Can't find either 'objdump' or 'ldd' to compute the dynamic library for '-l$1') fi
        Allen Wittenauer made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Allen Wittenauer added a comment -

        Hmm. Not quite right.

        Show
        Allen Wittenauer added a comment - Hmm. Not quite right.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12421172/hadoop-native-mac.patch
        against trunk revision 820094.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/71/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/71/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/71/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/71/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12421172/hadoop-native-mac.patch against trunk revision 820094. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/71/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/71/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/71/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/71/console This message is automatically generated.
        Allen Wittenauer made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Allen Wittenauer made changes -
        Attachment hadoop-native-mac.patch [ 12421172 ]
        Hide
        Allen Wittenauer added a comment -

        Fixes the grep z with grep $1

        Show
        Allen Wittenauer added a comment - Fixes the grep z with grep $1
        Hide
        Allen Wittenauer added a comment -

        Can we at least get this patch committed as is? Fixing aclocal.m4 is a step in the right direction. The rest can be fudged until we figure out what to do w/INCLUDE and friends.

        Also, at least with 1.6 on Leopard on my machine, I need to set LDFLAGS=-L/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Libraries .

        Show
        Allen Wittenauer added a comment - Can we at least get this patch committed as is? Fixing aclocal.m4 is a step in the right direction. The rest can be fudged until we figure out what to do w/INCLUDE and friends. Also, at least with 1.6 on Leopard on my machine, I need to set LDFLAGS=-L/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Libraries .
        Hide
        Allen Wittenauer added a comment - - edited

        For completeness, fink uses /sw as its root instead of /opt.

        Although we need to be wary about including a ton of paths. We might trigger NFS, which would be less than ideal.

        Is there some reason that we can't let the user set LDFLAGS, INCLUDE, etc, as env vars and suck them in during the config process? i.e., if I have my lzo set in a weirdo place (i.e., MacPorts, Fink, etc), then I should be able to:

        INCLUDE=/sw/include LDFLAGS=-L/sw/lib -R/swlib ant build

        ... or is ant not smart enough to pass those on?

        Show
        Allen Wittenauer added a comment - - edited For completeness, fink uses /sw as its root instead of /opt. Although we need to be wary about including a ton of paths. We might trigger NFS, which would be less than ideal. Is there some reason that we can't let the user set LDFLAGS, INCLUDE, etc, as env vars and suck them in during the config process? i.e., if I have my lzo set in a weirdo place (i.e., MacPorts, Fink, etc), then I should be able to: INCLUDE=/sw/include LDFLAGS=-L/sw/lib -R/swlib ant build ... or is ant not smart enough to pass those on?
        Hide
        Arun C Murthy added a comment -

        I guess that the configure script should probably support either install..

        Yes, and the configure script also needs to ensure that LDFLAGS also has the right path to find libjvm.so too...

        Show
        Arun C Murthy added a comment - I guess that the configure script should probably support either install.. Yes, and the configure script also needs to ensure that LDFLAGS also has the right path to find libjvm.so too...
        Hide
        Colin Evans added a comment -

        I installed the lzo package from http://www.oberhumer.com/opensource/lzo/ - the MacPorts lzo package drops its libs in /opt/local/include, which isn't in the default lib path. I guess that the configure script should probably support either install..

        Show
        Colin Evans added a comment - I installed the lzo package from http://www.oberhumer.com/opensource/lzo/ - the MacPorts lzo package drops its libs in /opt/local/include, which isn't in the default lib path. I guess that the configure script should probably support either install..
        Arun C Murthy made changes -
        Assignee Colin Evans [ colinhevans ]
        Hide
        Arun C Murthy added a comment -

        Colin, apologies that this has taken so long...

        I installed lzo2 on my Mac via macports and I needed to add '-I /opt/local/include' to configure.ac to get autoconf to find my lzo2 headers... how did you get around it without needing to modify configure.ac?

        Show
        Arun C Murthy added a comment - Colin, apologies that this has taken so long... I installed lzo2 on my Mac via macports and I needed to add '-I /opt/local/include' to configure.ac to get autoconf to find my lzo2 headers... how did you get around it without needing to modify configure.ac?
        Colin Evans made changes -
        Field Original Value New Value
        Attachment hadoop-native-mac.patch [ 12384863 ]
        Hide
        Colin Evans added a comment -

        A patch to acinclude.m4 that supports using otool to find a linked library.

        Show
        Colin Evans added a comment - A patch to acinclude.m4 that supports using otool to find a linked library.
        Colin Evans created issue -

          People

          • Assignee:
            Colin Evans
            Reporter:
            Colin Evans
          • Votes:
            1 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development