Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2127

mapreduce trunk builds are failing on hudson

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.203.1, 0.20.204.0, 0.23.0
    • Fix Version/s: 0.22.0, 0.23.0
    • Component/s: build, pipes
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/507/console

      [exec] checking for pthread.h... yes
      [exec] checking for pthread_create in -lpthread... yes
      [exec] checking for HMAC_Init in -lssl... no
      [exec] configure: error: Cannot find libssl.so
      [exec] /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/src/c++/pipes/configure: line 4250: exit: please: numeric argument required
      [exec] /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/src/c++/pipes/configure: line 4250: exit: please: numeric argument required

      BUILD FAILED
      /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/build.xml:1647: exec returned: 255

      1. MAPREDUCE-2127.patch
        0.7 kB
        Bruno Mahé
      2. MAPREDUCE-2127.patch
        1.08 MB
        Ranjit Mathew

        Activity

        Giridharan Kesavan created issue -
        Hide
        Ranjit Mathew added a comment -

        I see the same issue on Fedora 13 with ant -Dcompile.c++=true or with test-patch.sh. The
        error from the log-file in build/c++-build/Linux-i386-32/pipes/config.log is:

        configure:4176: checking for HMAC_Init in -lssl
        configure:4206: gcc -o conftest -g -O2 conftest.c -lssl -lpthread >&5
        /usr/bin/ld: /tmp/ccux6G7X.o: undefined reference to symbol 'HMAC_Init'
        /usr/bin/ld: note: 'HMAC_Init' is defined in DSO /lib/libcrypto.so.10 so try adding it to the linker command line
        /lib/libcrypto.so.10: could not read symbols: Invalid operation
        collect2: ld returned 1 exit status

        This is due to changes in ld (http://fedoraproject.org/wiki/UnderstandingDSOLinkChange) so
        that it no longer does implicit linking of shared libraries.

        See http://forum.nginx.org/read.php?30,133815,133869#msg-133869 for a possible solution.

        The issue was introduced with MAPREDUCE-1733.

        Show
        Ranjit Mathew added a comment - I see the same issue on Fedora 13 with ant -Dcompile.c++=true or with test-patch.sh . The error from the log-file in build/c++-build/Linux-i386-32/pipes/config.log is: configure:4176: checking for HMAC_Init in -lssl configure:4206: gcc -o conftest -g -O2 conftest.c -lssl -lpthread >&5 /usr/bin/ld: /tmp/ccux6G7X.o: undefined reference to symbol 'HMAC_Init' /usr/bin/ld: note: 'HMAC_Init' is defined in DSO /lib/libcrypto.so.10 so try adding it to the linker command line /lib/libcrypto.so.10: could not read symbols: Invalid operation collect2: ld returned 1 exit status This is due to changes in ld ( http://fedoraproject.org/wiki/UnderstandingDSOLinkChange ) so that it no longer does implicit linking of shared libraries. See http://forum.nginx.org/read.php?30,133815,133869#msg-133869 for a possible solution. The issue was introduced with MAPREDUCE-1733 .
        Ranjit Mathew made changes -
        Field Original Value New Value
        Component/s build [ 12312909 ]
        Component/s pipes [ 12312908 ]
        Hide
        Ranjit Mathew added a comment -

        Patch that fixes this issue. Tested on Fedora 13 (32-bit) and RHEL 4 U2 (64-bit).

        The "real" changes are to src/c++/utils/m4/hadoop_utils.m4. The rest were regenerated
        with autoreconf. I used (vanilla) GNU Autoconf 2.59, Automake 1.9.6 and Libtool 1.5.20
        on Fedora 13 to stay as close to the original files as possible.

        Show
        Ranjit Mathew added a comment - Patch that fixes this issue. Tested on Fedora 13 (32-bit) and RHEL 4 U2 (64-bit). The "real" changes are to src/c++/utils/m4/hadoop_utils.m4 . The rest were regenerated with autoreconf . I used (vanilla) GNU Autoconf 2.59, Automake 1.9.6 and Libtool 1.5.20 on Fedora 13 to stay as close to the original files as possible.
        Ranjit Mathew made changes -
        Attachment MAPREDUCE-2127.patch [ 12457058 ]
        Ranjit Mathew made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Giridharan Kesavan added a comment -

        I applied this patch and tested it on the same ubuntu machine that runs the build and here is the log:
        [exec] checking for unistd.h... yes
        [exec] checking pthread.h usability... yes
        [exec] checking pthread.h presence... yes
        [exec] checking for pthread.h... yes
        [exec] checking for pthread_create in -lpthread... yes
        [exec] checking for HMAC_Init in -lcrypto... no
        [exec] configure: error: Cannot find libcrypto.so
        [exec] trunk/src/c++/pipes/configure: line 4250: exit: please: numeric argument required
        [exec] trunk/src/c++/pipes/configure: line 4250: exit: please: numeric argument required

        Show
        Giridharan Kesavan added a comment - I applied this patch and tested it on the same ubuntu machine that runs the build and here is the log: [exec] checking for unistd.h... yes [exec] checking pthread.h usability... yes [exec] checking pthread.h presence... yes [exec] checking for pthread.h... yes [exec] checking for pthread_create in -lpthread... yes [exec] checking for HMAC_Init in -lcrypto... no [exec] configure: error: Cannot find libcrypto.so [exec] trunk/src/c++/pipes/configure: line 4250: exit: please: numeric argument required [exec] trunk/src/c++/pipes/configure: line 4250: exit: please: numeric argument required
        Hide
        Andrew Milkowski added a comment -

        on Fedora 14

        [hadoop@localhost pipes]$ uname -r
        2.6.35.6-43.fc14.x86_64

        patch worked (thanks Ranjit)

        1. applied patch just to src/c++/utils/m4/hadoop_utils.m4
        2. in src/c++/pipes

        autoreconf
        make distclean

        ant compile -Dversion="0.20.2" -Dcompile.c++=true -Dlibhdfs=true

        r.;

        Show
        Andrew Milkowski added a comment - on Fedora 14 [hadoop@localhost pipes] $ uname -r 2.6.35.6-43.fc14.x86_64 patch worked (thanks Ranjit) 1. applied patch just to src/c++/utils/m4/hadoop_utils.m4 2. in src/c++/pipes autoreconf make distclean ant compile -Dversion="0.20.2" -Dcompile.c++=true -Dlibhdfs=true r.;
        Hide
        Ranjit Mathew added a comment -

        Giri, do you have openssl-devel installed? If yes, what does "rpm -ql openssl-devel | grep ^/usr/lib" say?
        If not, you have to have it installed to compile the Pipes C++ code.

        Show
        Ranjit Mathew added a comment - Giri, do you have openssl-devel installed? If yes, what does " rpm -ql openssl-devel | grep ^/usr/lib " say? If not, you have to have it installed to compile the Pipes C++ code.
        Hide
        Giridharan Kesavan added a comment -

        The mr trunk builds used to run on this same h3 machine all this while.
        Perhaps openssl-devel dependency got introduced recently?
        Anyways, installing openssl package on the ubuntu build machine fixed the failure with the patch.

        Thanks Ranjit

        Show
        Giridharan Kesavan added a comment - The mr trunk builds used to run on this same h3 machine all this while. Perhaps openssl-devel dependency got introduced recently? Anyways, installing openssl package on the ubuntu build machine fixed the failure with the patch. Thanks Ranjit
        Hide
        Giridharan Kesavan added a comment -

        +1 , I would commit this patch in a while and get the trunk build going..

        Show
        Giridharan Kesavan added a comment - +1 , I would commit this patch in a while and get the trunk build going..
        Hide
        Giridharan Kesavan added a comment -

        Just by installing the missing package (libssl-dev) pipes is getting compiled without any issues on the ubuntu build servers and I never have to apply this patch.

        Show
        Giridharan Kesavan added a comment - Just by installing the missing package (libssl-dev) pipes is getting compiled without any issues on the ubuntu build servers and I never have to apply this patch.
        Hide
        Ranjit Mathew added a comment -

        Just by installing the missing package (libssl-dev) pipes is getting compiled without any issues on the ubuntu build servers and I never have to apply this patch.

        Yes, but at least on Fedora 13, this patch is required due to the linker changes I referred to earlier.

        Show
        Ranjit Mathew added a comment - Just by installing the missing package (libssl-dev) pipes is getting compiled without any issues on the ubuntu build servers and I never have to apply this patch. Yes, but at least on Fedora 13, this patch is required due to the linker changes I referred to earlier.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12457058/MAPREDUCE-2127.patch
        against trunk revision 1074251.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/59//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12457058/MAPREDUCE-2127.patch against trunk revision 1074251. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/59//console This message is automatically generated.
        Ranjit Mathew made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Bruno Mahé added a comment -

        Here is a reworked patch to make it compile on fedora 15.

        It was tested and working on Centos 5, fedora 15 and ubuntu maverick 10.10.

        Show
        Bruno Mahé added a comment - Here is a reworked patch to make it compile on fedora 15. It was tested and working on Centos 5, fedora 15 and ubuntu maverick 10.10.
        Bruno Mahé made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Affects Version/s 0.20.203.1 [ 12316291 ]
        Affects Version/s 0.20.204.0 [ 12316318 ]
        Affects Version/s 0.23.0 [ 12315570 ]
        Hide
        Bruno Mahé added a comment -

        HMAC_Init seems to be defined in libcrypto and fedora policy regarding DSO linking prevent it from being picked up as explainied a few comments above.

        Show
        Bruno Mahé added a comment - HMAC_Init seems to be defined in libcrypto and fedora policy regarding DSO linking prevent it from being picked up as explainied a few comments above.
        Bruno Mahé made changes -
        Attachment MAPREDUCE-2127.patch [ 12482904 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12482904/MAPREDUCE-2127.patch
        against trunk revision 1136261.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:
        org.apache.hadoop.cli.TestMRCLI
        org.apache.hadoop.fs.TestFileSystem

        -1 contrib tests. The patch failed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/403//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/403//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/403//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12482904/MAPREDUCE-2127.patch against trunk revision 1136261. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI org.apache.hadoop.fs.TestFileSystem -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/403//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/403//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/403//console This message is automatically generated.
        Hide
        Todd Lipcon added a comment -

        +1, thanks Bruno!

        Show
        Todd Lipcon added a comment - +1, thanks Bruno!
        Hide
        Todd Lipcon added a comment -

        I'm checking with Nigel to see if we can put this in 0.22 as well, since it's just a build fix and the current issue blocks compilation on some platforms.

        Show
        Todd Lipcon added a comment - I'm checking with Nigel to see if we can put this in 0.22 as well, since it's just a build fix and the current issue blocks compilation on some platforms.
        Todd Lipcon made changes -
        Assignee Bruno Mahé [ bmahe ]
        Hide
        Nigel Daley added a comment -

        +1 for 0.22

        Show
        Nigel Daley added a comment - +1 for 0.22
        Eli Collins made changes -
        Hadoop Flags [Reviewed]
        Fix Version/s 0.23.0 [ 12315570 ]
        Hide
        Eli Collins added a comment -

        I've committed this. Thanks Bruno!

        Show
        Eli Collins added a comment - I've committed this. Thanks Bruno!
        Eli Collins made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Eli Collins added a comment -

        Merged to branch-22 as well.

        Show
        Eli Collins added a comment - Merged to branch-22 as well.
        Eli Collins made changes -
        Fix Version/s 0.22.0 [ 12314184 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #760 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/760/)
        MAPREDUCE-2127. mapreduce trunk builds are filing on hudson. Contributed by Bruno Mahé

        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1152068
        Files :

        • /hadoop/common/trunk/mapreduce/CHANGES.txt
        • /hadoop/common/trunk/mapreduce/src/c++/utils/m4/hadoop_utils.m4
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #760 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/760/ ) MAPREDUCE-2127 . mapreduce trunk builds are filing on hudson. Contributed by Bruno Mahé eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1152068 Files : /hadoop/common/trunk/mapreduce/CHANGES.txt /hadoop/common/trunk/mapreduce/src/c++/utils/m4/hadoop_utils.m4
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #751 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/751/)
        MAPREDUCE-2127. mapreduce trunk builds are filing on hudson. Contributed by Bruno Mahé

        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1152068
        Files :

        • /hadoop/common/trunk/mapreduce/CHANGES.txt
        • /hadoop/common/trunk/mapreduce/src/c++/utils/m4/hadoop_utils.m4
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #751 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/751/ ) MAPREDUCE-2127 . mapreduce trunk builds are filing on hudson. Contributed by Bruno Mahé eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1152068 Files : /hadoop/common/trunk/mapreduce/CHANGES.txt /hadoop/common/trunk/mapreduce/src/c++/utils/m4/hadoop_utils.m4
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-22-branch #62 (See https://builds.apache.org/job/Hadoop-Mapreduce-22-branch/62/)
        MAPREDUCE-2127. svn merge -c 1152068 from trunk

        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1152070
        Files :

        • /hadoop/common/branches/branch-0.22/mapreduce/src/c++/utils/m4/hadoop_utils.m4
        • /hadoop/common/branches/branch-0.22/mapreduce/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-22-branch #62 (See https://builds.apache.org/job/Hadoop-Mapreduce-22-branch/62/ ) MAPREDUCE-2127 . svn merge -c 1152068 from trunk eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1152070 Files : /hadoop/common/branches/branch-0.22/mapreduce/src/c++/utils/m4/hadoop_utils.m4 /hadoop/common/branches/branch-0.22/mapreduce/CHANGES.txt
        Matt Foley made changes -
        Summary mapreduce trunk builds are filing on hudson .. mapreduce trunk builds are failing on hudson
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Bruno Mahé
            Reporter:
            Giridharan Kesavan
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development