Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2127

mapreduce trunk builds are failing on hudson

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.203.1, 0.20.204.0, 0.23.0
    • Fix Version/s: 0.22.0, 0.23.0
    • Component/s: build, pipes
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/507/console

      [exec] checking for pthread.h... yes
      [exec] checking for pthread_create in -lpthread... yes
      [exec] checking for HMAC_Init in -lssl... no
      [exec] configure: error: Cannot find libssl.so
      [exec] /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/src/c++/pipes/configure: line 4250: exit: please: numeric argument required
      [exec] /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/src/c++/pipes/configure: line 4250: exit: please: numeric argument required

      BUILD FAILED
      /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/build.xml:1647: exec returned: 255

      1. MAPREDUCE-2127.patch
        1.08 MB
        Ranjit Mathew
      2. MAPREDUCE-2127.patch
        0.7 kB
        Bruno Mahé

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-22-branch #62 (See https://builds.apache.org/job/Hadoop-Mapreduce-22-branch/62/)
        MAPREDUCE-2127. svn merge -c 1152068 from trunk

        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1152070
        Files :

        • /hadoop/common/branches/branch-0.22/mapreduce/src/c++/utils/m4/hadoop_utils.m4
        • /hadoop/common/branches/branch-0.22/mapreduce/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-22-branch #62 (See https://builds.apache.org/job/Hadoop-Mapreduce-22-branch/62/ ) MAPREDUCE-2127 . svn merge -c 1152068 from trunk eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1152070 Files : /hadoop/common/branches/branch-0.22/mapreduce/src/c++/utils/m4/hadoop_utils.m4 /hadoop/common/branches/branch-0.22/mapreduce/CHANGES.txt
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #751 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/751/)
        MAPREDUCE-2127. mapreduce trunk builds are filing on hudson. Contributed by Bruno Mahé

        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1152068
        Files :

        • /hadoop/common/trunk/mapreduce/CHANGES.txt
        • /hadoop/common/trunk/mapreduce/src/c++/utils/m4/hadoop_utils.m4
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #751 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/751/ ) MAPREDUCE-2127 . mapreduce trunk builds are filing on hudson. Contributed by Bruno Mahé eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1152068 Files : /hadoop/common/trunk/mapreduce/CHANGES.txt /hadoop/common/trunk/mapreduce/src/c++/utils/m4/hadoop_utils.m4
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #760 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/760/)
        MAPREDUCE-2127. mapreduce trunk builds are filing on hudson. Contributed by Bruno Mahé

        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1152068
        Files :

        • /hadoop/common/trunk/mapreduce/CHANGES.txt
        • /hadoop/common/trunk/mapreduce/src/c++/utils/m4/hadoop_utils.m4
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #760 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/760/ ) MAPREDUCE-2127 . mapreduce trunk builds are filing on hudson. Contributed by Bruno Mahé eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1152068 Files : /hadoop/common/trunk/mapreduce/CHANGES.txt /hadoop/common/trunk/mapreduce/src/c++/utils/m4/hadoop_utils.m4
        Hide
        Eli Collins added a comment -

        Merged to branch-22 as well.

        Show
        Eli Collins added a comment - Merged to branch-22 as well.
        Hide
        Eli Collins added a comment -

        I've committed this. Thanks Bruno!

        Show
        Eli Collins added a comment - I've committed this. Thanks Bruno!
        Hide
        Nigel Daley added a comment -

        +1 for 0.22

        Show
        Nigel Daley added a comment - +1 for 0.22
        Hide
        Todd Lipcon added a comment -

        I'm checking with Nigel to see if we can put this in 0.22 as well, since it's just a build fix and the current issue blocks compilation on some platforms.

        Show
        Todd Lipcon added a comment - I'm checking with Nigel to see if we can put this in 0.22 as well, since it's just a build fix and the current issue blocks compilation on some platforms.
        Hide
        Todd Lipcon added a comment -

        +1, thanks Bruno!

        Show
        Todd Lipcon added a comment - +1, thanks Bruno!
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12482904/MAPREDUCE-2127.patch
        against trunk revision 1136261.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:
        org.apache.hadoop.cli.TestMRCLI
        org.apache.hadoop.fs.TestFileSystem

        -1 contrib tests. The patch failed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/403//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/403//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/403//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12482904/MAPREDUCE-2127.patch against trunk revision 1136261. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI org.apache.hadoop.fs.TestFileSystem -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/403//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/403//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/403//console This message is automatically generated.
        Hide
        Bruno Mahé added a comment -

        HMAC_Init seems to be defined in libcrypto and fedora policy regarding DSO linking prevent it from being picked up as explainied a few comments above.

        Show
        Bruno Mahé added a comment - HMAC_Init seems to be defined in libcrypto and fedora policy regarding DSO linking prevent it from being picked up as explainied a few comments above.
        Hide
        Bruno Mahé added a comment -

        Here is a reworked patch to make it compile on fedora 15.

        It was tested and working on Centos 5, fedora 15 and ubuntu maverick 10.10.

        Show
        Bruno Mahé added a comment - Here is a reworked patch to make it compile on fedora 15. It was tested and working on Centos 5, fedora 15 and ubuntu maverick 10.10.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12457058/MAPREDUCE-2127.patch
        against trunk revision 1074251.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/59//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12457058/MAPREDUCE-2127.patch against trunk revision 1074251. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/59//console This message is automatically generated.
        Hide
        Ranjit Mathew added a comment -

        Just by installing the missing package (libssl-dev) pipes is getting compiled without any issues on the ubuntu build servers and I never have to apply this patch.

        Yes, but at least on Fedora 13, this patch is required due to the linker changes I referred to earlier.

        Show
        Ranjit Mathew added a comment - Just by installing the missing package (libssl-dev) pipes is getting compiled without any issues on the ubuntu build servers and I never have to apply this patch. Yes, but at least on Fedora 13, this patch is required due to the linker changes I referred to earlier.
        Hide
        Giridharan Kesavan added a comment -

        Just by installing the missing package (libssl-dev) pipes is getting compiled without any issues on the ubuntu build servers and I never have to apply this patch.

        Show
        Giridharan Kesavan added a comment - Just by installing the missing package (libssl-dev) pipes is getting compiled without any issues on the ubuntu build servers and I never have to apply this patch.
        Hide
        Giridharan Kesavan added a comment -

        +1 , I would commit this patch in a while and get the trunk build going..

        Show
        Giridharan Kesavan added a comment - +1 , I would commit this patch in a while and get the trunk build going..
        Hide
        Giridharan Kesavan added a comment -

        The mr trunk builds used to run on this same h3 machine all this while.
        Perhaps openssl-devel dependency got introduced recently?
        Anyways, installing openssl package on the ubuntu build machine fixed the failure with the patch.

        Thanks Ranjit

        Show
        Giridharan Kesavan added a comment - The mr trunk builds used to run on this same h3 machine all this while. Perhaps openssl-devel dependency got introduced recently? Anyways, installing openssl package on the ubuntu build machine fixed the failure with the patch. Thanks Ranjit
        Hide
        Ranjit Mathew added a comment -

        Giri, do you have openssl-devel installed? If yes, what does "rpm -ql openssl-devel | grep ^/usr/lib" say?
        If not, you have to have it installed to compile the Pipes C++ code.

        Show
        Ranjit Mathew added a comment - Giri, do you have openssl-devel installed? If yes, what does " rpm -ql openssl-devel | grep ^/usr/lib " say? If not, you have to have it installed to compile the Pipes C++ code.
        Hide
        Andrew Milkowski added a comment -

        on Fedora 14

        [hadoop@localhost pipes]$ uname -r
        2.6.35.6-43.fc14.x86_64

        patch worked (thanks Ranjit)

        1. applied patch just to src/c++/utils/m4/hadoop_utils.m4
        2. in src/c++/pipes

        autoreconf
        make distclean

        ant compile -Dversion="0.20.2" -Dcompile.c++=true -Dlibhdfs=true

        r.;

        Show
        Andrew Milkowski added a comment - on Fedora 14 [hadoop@localhost pipes] $ uname -r 2.6.35.6-43.fc14.x86_64 patch worked (thanks Ranjit) 1. applied patch just to src/c++/utils/m4/hadoop_utils.m4 2. in src/c++/pipes autoreconf make distclean ant compile -Dversion="0.20.2" -Dcompile.c++=true -Dlibhdfs=true r.;
        Hide
        Giridharan Kesavan added a comment -

        I applied this patch and tested it on the same ubuntu machine that runs the build and here is the log:
        [exec] checking for unistd.h... yes
        [exec] checking pthread.h usability... yes
        [exec] checking pthread.h presence... yes
        [exec] checking for pthread.h... yes
        [exec] checking for pthread_create in -lpthread... yes
        [exec] checking for HMAC_Init in -lcrypto... no
        [exec] configure: error: Cannot find libcrypto.so
        [exec] trunk/src/c++/pipes/configure: line 4250: exit: please: numeric argument required
        [exec] trunk/src/c++/pipes/configure: line 4250: exit: please: numeric argument required

        Show
        Giridharan Kesavan added a comment - I applied this patch and tested it on the same ubuntu machine that runs the build and here is the log: [exec] checking for unistd.h... yes [exec] checking pthread.h usability... yes [exec] checking pthread.h presence... yes [exec] checking for pthread.h... yes [exec] checking for pthread_create in -lpthread... yes [exec] checking for HMAC_Init in -lcrypto... no [exec] configure: error: Cannot find libcrypto.so [exec] trunk/src/c++/pipes/configure: line 4250: exit: please: numeric argument required [exec] trunk/src/c++/pipes/configure: line 4250: exit: please: numeric argument required
        Hide
        Ranjit Mathew added a comment -

        Patch that fixes this issue. Tested on Fedora 13 (32-bit) and RHEL 4 U2 (64-bit).

        The "real" changes are to src/c++/utils/m4/hadoop_utils.m4. The rest were regenerated
        with autoreconf. I used (vanilla) GNU Autoconf 2.59, Automake 1.9.6 and Libtool 1.5.20
        on Fedora 13 to stay as close to the original files as possible.

        Show
        Ranjit Mathew added a comment - Patch that fixes this issue. Tested on Fedora 13 (32-bit) and RHEL 4 U2 (64-bit). The "real" changes are to src/c++/utils/m4/hadoop_utils.m4 . The rest were regenerated with autoreconf . I used (vanilla) GNU Autoconf 2.59, Automake 1.9.6 and Libtool 1.5.20 on Fedora 13 to stay as close to the original files as possible.
        Hide
        Ranjit Mathew added a comment -

        I see the same issue on Fedora 13 with ant -Dcompile.c++=true or with test-patch.sh. The
        error from the log-file in build/c++-build/Linux-i386-32/pipes/config.log is:

        configure:4176: checking for HMAC_Init in -lssl
        configure:4206: gcc -o conftest -g -O2 conftest.c -lssl -lpthread >&5
        /usr/bin/ld: /tmp/ccux6G7X.o: undefined reference to symbol 'HMAC_Init'
        /usr/bin/ld: note: 'HMAC_Init' is defined in DSO /lib/libcrypto.so.10 so try adding it to the linker command line
        /lib/libcrypto.so.10: could not read symbols: Invalid operation
        collect2: ld returned 1 exit status

        This is due to changes in ld (http://fedoraproject.org/wiki/UnderstandingDSOLinkChange) so
        that it no longer does implicit linking of shared libraries.

        See http://forum.nginx.org/read.php?30,133815,133869#msg-133869 for a possible solution.

        The issue was introduced with MAPREDUCE-1733.

        Show
        Ranjit Mathew added a comment - I see the same issue on Fedora 13 with ant -Dcompile.c++=true or with test-patch.sh . The error from the log-file in build/c++-build/Linux-i386-32/pipes/config.log is: configure:4176: checking for HMAC_Init in -lssl configure:4206: gcc -o conftest -g -O2 conftest.c -lssl -lpthread >&5 /usr/bin/ld: /tmp/ccux6G7X.o: undefined reference to symbol 'HMAC_Init' /usr/bin/ld: note: 'HMAC_Init' is defined in DSO /lib/libcrypto.so.10 so try adding it to the linker command line /lib/libcrypto.so.10: could not read symbols: Invalid operation collect2: ld returned 1 exit status This is due to changes in ld ( http://fedoraproject.org/wiki/UnderstandingDSOLinkChange ) so that it no longer does implicit linking of shared libraries. See http://forum.nginx.org/read.php?30,133815,133869#msg-133869 for a possible solution. The issue was introduced with MAPREDUCE-1733 .

          People

          • Assignee:
            Bruno Mahé
            Reporter:
            Giridharan Kesavan
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development