Hadoop Common
  1. Hadoop Common
  2. HADOOP-7868

Hadoop native fails to compile when default linker option is -Wl,--as-needed

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0, 2.0.0-alpha
    • Fix Version/s: 1.2.0, 2.0.2-alpha, 0.23.6
    • Component/s: native
    • Labels:
      None
    • Environment:

      Ubuntu Precise, Ubuntu Oneiric, Debian Unstable

    • Hadoop Flags:
      Reviewed

      Description

      Recent releases of Ubuntu and Debian have switched to using --as-needed as default when linking binaries.

      As a result the AC_COMPUTE_NEEDED_DSO fails to find the required DSO names during execution of configure resulting in a build failure.

      Explicitly using "Wl,-no-as-needed" in this macro when required resolves this issue.

      See http://wiki.debian.org/ToolChain/DSOLinking for a few more details

      1. HADOOP-7868.patch
        2 kB
        James Page
      2. HADOOP-7868-portable.patch
        3 kB
        Trevor Robinson
      3. hadoop-7868-b1.txt
        3 kB
        Eli Collins

        Issue Links

          Activity

          Hide
          James Page added a comment -

          Initial patch attached for review; I'm no autoconf expert so this might not be that elegant.

          Show
          James Page added a comment - Initial patch attached for review; I'm no autoconf expert so this might not be that elegant.
          Hide
          Mark Butler added a comment -
          Show
          Mark Butler added a comment - This is the same issue as https://issues.apache.org/jira/browse/HADOOP-7979
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12505473/HADOOP-7868.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/662//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/662//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12505473/HADOOP-7868.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/662//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/662//console This message is automatically generated.
          Hide
          Eugene Koontz added a comment -

          +1 works well for me. My only concern would be, what about other linkers (for example older versions of GNU ld or other brands of linkers) that do not support --no-as-needed - would they throw an error?

          Show
          Eugene Koontz added a comment - +1 works well for me. My only concern would be, what about other linkers (for example older versions of GNU ld or other brands of linkers) that do not support --no-as-needed - would they throw an error?
          Hide
          Daryn Sharp added a comment -

          Yes, linkers will error if the flag isn't supported. The problem is really in the library check function, not the linker. Other open source packages typically compile a test program that:

          • includes a header for the library
          • externs a function or variable from the library
          • main tries to reference/call the extern-ed thing

          I think that's the better fix, versus trying to stop the compiler from doing the right thing.

          Show
          Daryn Sharp added a comment - Yes, linkers will error if the flag isn't supported. The problem is really in the library check function, not the linker. Other open source packages typically compile a test program that: includes a header for the library externs a function or variable from the library main tries to reference/call the extern-ed thing I think that's the better fix, versus trying to stop the compiler from doing the right thing.
          Hide
          Trevor Robinson added a comment -

          This patch fixes the issue using the approach suggested by Daryn: it outputs configure test code that makes use of the library being detected for both zlib and snappy.

          Show
          Trevor Robinson added a comment - This patch fixes the issue using the approach suggested by Daryn: it outputs configure test code that makes use of the library being detected for both zlib and snappy.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12525955/HADOOP-7868-portable.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/958//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/958//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525955/HADOOP-7868-portable.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/958//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/958//console This message is automatically generated.
          Hide
          Daryn Sharp added a comment -

          +1 Looks good! I'm not sure if the objdump check is still necessary, but it probably doesn't hurt.

          Show
          Daryn Sharp added a comment - +1 Looks good! I'm not sure if the objdump check is still necessary, but it probably doesn't hurt.
          Hide
          Trevor Robinson added a comment -

          I was surprised that supporting three different tools was necessary, but I wasn't bold enough to assume it was safe to remove any.

          As a bit of context for someone thinking about committing this patch (please do!), it along with HADOOP-8370 and HDFS-3383 enabling building on Ubuntu 12.04 ARM Server.

          Show
          Trevor Robinson added a comment - I was surprised that supporting three different tools was necessary, but I wasn't bold enough to assume it was safe to remove any. As a bit of context for someone thinking about committing this patch (please do!), it along with HADOOP-8370 and HDFS-3383 enabling building on Ubuntu 12.04 ARM Server.
          Hide
          James Page added a comment -

          +1 Works for me

          (and +1 to including the other two patches that Trevor identified for building on ARM server).

          Show
          James Page added a comment - +1 Works for me (and +1 to including the other two patches that Trevor identified for building on ARM server).
          Hide
          Colin Patrick McCabe added a comment -

          +1 Works for me.

          Looks reasonable.

          Show
          Colin Patrick McCabe added a comment - +1 Works for me. Looks reasonable.
          Hide
          Eli Collins added a comment -

          +1 looks good. I tested the native build with this patch on Ubuntu 12.04, 10.10 and Centos 5.5.

          Show
          Eli Collins added a comment - +1 looks good. I tested the native build with this patch on Ubuntu 12.04, 10.10 and Centos 5.5.
          Hide
          Eli Collins added a comment -

          Thanks guys. I've committed this to trunk and merged to branch-2 and branch-2-alpha.

          Show
          Eli Collins added a comment - Thanks guys. I've committed this to trunk and merged to branch-2 and branch-2-alpha.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2218 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2218/)
          HADOOP-7868. Hadoop native fails to compile when default linker option is Wl,-as-needed. Contributed by Trevor Robinson (Revision 1336491)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1336491
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/acinclude.m4
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/configure.ac
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2218 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2218/ ) HADOOP-7868 . Hadoop native fails to compile when default linker option is Wl, -as-needed. Contributed by Trevor Robinson (Revision 1336491) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1336491 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/acinclude.m4 /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/configure.ac
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2293 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2293/)
          HADOOP-7868. Hadoop native fails to compile when default linker option is Wl,-as-needed. Contributed by Trevor Robinson (Revision 1336491)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1336491
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/acinclude.m4
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/configure.ac
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2293 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2293/ ) HADOOP-7868 . Hadoop native fails to compile when default linker option is Wl, -as-needed. Contributed by Trevor Robinson (Revision 1336491) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1336491 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/acinclude.m4 /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/configure.ac
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2235 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2235/)
          HADOOP-7868. Hadoop native fails to compile when default linker option is Wl,-as-needed. Contributed by Trevor Robinson (Revision 1336491)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1336491
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/acinclude.m4
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/configure.ac
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2235 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2235/ ) HADOOP-7868 . Hadoop native fails to compile when default linker option is Wl, -as-needed. Contributed by Trevor Robinson (Revision 1336491) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1336491 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/acinclude.m4 /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/configure.ac
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1075 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1075/)
          HADOOP-7868. Hadoop native fails to compile when default linker option is Wl,-as-needed. Contributed by Trevor Robinson (Revision 1336491)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1336491
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/acinclude.m4
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/configure.ac
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1075 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1075/ ) HADOOP-7868 . Hadoop native fails to compile when default linker option is Wl, -as-needed. Contributed by Trevor Robinson (Revision 1336491) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1336491 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/acinclude.m4 /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/configure.ac
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1040 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1040/)
          HADOOP-7868. Hadoop native fails to compile when default linker option is Wl,-as-needed. Contributed by Trevor Robinson (Revision 1336491)

          Result = FAILURE
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1336491
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/acinclude.m4
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/configure.ac
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1040 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1040/ ) HADOOP-7868 . Hadoop native fails to compile when default linker option is Wl, -as-needed. Contributed by Trevor Robinson (Revision 1336491) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1336491 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/acinclude.m4 /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/configure.ac
          Hide
          Daryn Sharp added a comment -

          Should this jira be closed, or is something pending?

          Show
          Daryn Sharp added a comment - Should this jira be closed, or is something pending?
          Hide
          Eli Collins added a comment -

          Left it open because there's a 1.1.0 target version and there's no branch-1 patch yet.

          Show
          Eli Collins added a comment - Left it open because there's a 1.1.0 target version and there's no branch-1 patch yet.
          Hide
          Eli Collins added a comment -

          Same patch just different paths also works for branch-1. I verified compile-native with the attached patch on 12.04, will merge this.

          Show
          Eli Collins added a comment - Same patch just different paths also works for branch-1. I verified compile-native with the attached patch on 12.04, will merge this.
          Hide
          Eli Collins added a comment -

          I've committed this to branch-1.

          Show
          Eli Collins added a comment - I've committed this to branch-1.
          Hide
          Thomas Graves added a comment -

          I pulled this into branch-0.23

          Show
          Thomas Graves added a comment - I pulled this into branch-0.23
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #449 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/449/)
          HADOOP-7868. Hadoop native fails to compile when default linker option is Wl,-as-needed. (Trevor Robinson via tgraves) (Revision 1414370)

          Result = SUCCESS
          tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1414370
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/acinclude.m4
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/configure.ac
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #449 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/449/ ) HADOOP-7868 . Hadoop native fails to compile when default linker option is Wl, -as-needed. (Trevor Robinson via tgraves) (Revision 1414370) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1414370 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/acinclude.m4 /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/configure.ac

            People

            • Assignee:
              Trevor Robinson
              Reporter:
              James Page
            • Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development