Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.3
    • Fix Version/s: 0.23.0
    • Component/s: build
    • Labels:
      None
    • Environment:

      Redhat 5.5, Java 6

    • Hadoop Flags:
      Reviewed

      Description

      Post MAPRED-2521 ant binary fails without "-Dcompile.c+=true -Dcompile.native=true". The bin-package is trying to copy from the c+ lib dir which doesn't exist yet. The binary target should check for the existence of this dir or would also be reasonable to depend on the compile-c++ (since this is the binary target).

      1. MAPREDUCE-2559.patch
        0.9 kB
        Eric Yang
      2. MAPREDUCE-2559-1.patch
        2 kB
        Eric Yang
      3. MAPREDUCE-2559-2.patch
        1 kB
        Eric Yang
      4. MAPREDUCE-2559-3.patch
        0.7 kB
        Eric Yang
      5. mapreduce-2559-4.patch
        0.8 kB
        Eli Collins

        Issue Links

          Activity

          Eric Yang created issue -
          Eric Yang made changes -
          Field Original Value New Value
          Link This issue relates to MAPREDUCE-2521 [ MAPREDUCE-2521 ]
          Owen O'Malley made changes -
          Link This issue relates to HDFS-2022 [ HDFS-2022 ]
          Hide
          Eric Yang added a comment -

          Enforce binary target to depends on compile.c++.

          Show
          Eric Yang added a comment - Enforce binary target to depends on compile.c++.
          Eric Yang made changes -
          Attachment MAPREDUCE-2559.patch [ 12481258 ]
          Eric Yang made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Assignee Eric Yang [ eyang ]
          Hide
          Eli Collins added a comment -

          Shouldn't the binary target build all the c++ targets, including task-contoller and librecordio? IIUC this will just build pipes, utils and examples.

          Show
          Eli Collins added a comment - Shouldn't the binary target build all the c++ targets, including task-contoller and librecordio? IIUC this will just build pipes, utils and examples.
          Hide
          Eric Yang added a comment -

          librecordio requires xerces. For librecordio to compile, path to xerces must be defined in build.properties. I am trying to make "ant binary" as new comer proof as possible as suggested by Amar Kamat in MAPREDUCE-2521. Hence, task-controller and librecordio are not included to lower the barrier of entry. It would be easy to include librecordio and task-controller, if more people comments on this jira that is the right thing to do.

          Show
          Eric Yang added a comment - librecordio requires xerces. For librecordio to compile, path to xerces must be defined in build.properties. I am trying to make "ant binary" as new comer proof as possible as suggested by Amar Kamat in MAPREDUCE-2521 . Hence, task-controller and librecordio are not included to lower the barrier of entry. It would be easy to include librecordio and task-controller, if more people comments on this jira that is the right thing to do.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12481258/MAPREDUCE-2559.patch
          against trunk revision 1130554.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.cli.TestMRCLI

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/335//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/335//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/335//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481258/MAPREDUCE-2559.patch against trunk revision 1130554. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/335//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/335//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/335//console This message is automatically generated.
          Hide
          Eric Yang added a comment -

          Option #1, ant binary must provide -D for a proper build, or the build fails with this message:

          check-xercescroot:
          
          BUILD FAILED
          /home/eyang/sandbox/hadoop/git/hadoop-mapreduce/build.xml:1826: 'xercescroot' is not defined.  Please pass -Dxercescroot=<base of xerces 2.8.0 installation> to Ant on the command-line.
          
          Show
          Eric Yang added a comment - Option #1, ant binary must provide -D for a proper build, or the build fails with this message: check-xercescroot: BUILD FAILED /home/eyang/sandbox/hadoop/git/hadoop-mapreduce/build.xml:1826: 'xercescroot' is not defined. Please pass -Dxercescroot=<base of xerces 2.8.0 installation> to Ant on the command-line.
          Eric Yang made changes -
          Attachment MAPREDUCE-2559-1.patch [ 12481283 ]
          Hide
          Eric Yang added a comment -

          Sorry, incompleted statement in -D in previous message. Usage:

          ant clean binary -Dxercescroot=/path/to/xercescroot
          
          Show
          Eric Yang added a comment - Sorry, incompleted statement in -D in previous message. Usage: ant clean binary -Dxercescroot=/path/to/xercescroot
          Hide
          Eric Yang added a comment -

          Option #2, "ant binary" compiles pipes, and task-controller and utils. librecordio is not completed unless -Dlibrecordio and -Dxercescroot is defined.

          At this point, it's a pick and choose.

          Show
          Eric Yang added a comment - Option #2, "ant binary" compiles pipes, and task-controller and utils. librecordio is not completed unless -Dlibrecordio and -Dxercescroot is defined. At this point, it's a pick and choose.
          Eric Yang made changes -
          Attachment MAPREDUCE-2559-2.patch [ 12481285 ]
          Hide
          Todd Lipcon added a comment -

          I prefer to keep it as it was before – if you don't specify -Dcompile.c++=1 and -Dcompile.native=1, it didn't compile any native code, right?

          Strongly against the option that requires xerces, since librecordio is used only by a very tiny subset of users, and is as far as I know deprecated for quite some time.

          Show
          Todd Lipcon added a comment - I prefer to keep it as it was before – if you don't specify -Dcompile.c++=1 and -Dcompile.native=1, it didn't compile any native code, right? Strongly against the option that requires xerces, since librecordio is used only by a very tiny subset of users, and is as far as I know deprecated for quite some time.
          Hide
          Eric Yang added a comment -

          In HDFS-2022, HDFS community decided to change binary target to depends on libhdfs. Hence, it may be good to be consistent that binary target compiles c++ and native code. However, here is another patch that will avoid compile of any c++ code for binary target unless specified -Dcompile.* flags. Just in case.

          Show
          Eric Yang added a comment - In HDFS-2022 , HDFS community decided to change binary target to depends on libhdfs. Hence, it may be good to be consistent that binary target compiles c++ and native code. However, here is another patch that will avoid compile of any c++ code for binary target unless specified -Dcompile.* flags. Just in case.
          Eric Yang made changes -
          Attachment MAPREDUCE-2559-3.patch [ 12481297 ]
          Hide
          Eli Collins added a comment -

          Agree that we should not automatically compile librecordio since it does not build out of the box (we don't bundle it's dependency). I'm fine if MR wants to keep the compilation contigent on compile.c++ being passed (note that compile.native is just in common), though given that ant binary is supposed to produce a binary tarball it seems like it should build all the binaries. Is the concern that there should be a target that produces a tarball w/ just java code? Newcomers can always use ant jar to build just the java code (assuming they don't need a tarball).

          Show
          Eli Collins added a comment - Agree that we should not automatically compile librecordio since it does not build out of the box (we don't bundle it's dependency). I'm fine if MR wants to keep the compilation contigent on compile.c++ being passed (note that compile.native is just in common), though given that ant binary is supposed to produce a binary tarball it seems like it should build all the binaries. Is the concern that there should be a target that produces a tarball w/ just java code? Newcomers can always use ant jar to build just the java code (assuming they don't need a tarball).
          Hide
          Todd Lipcon added a comment -

          For doing a full release binary tarball, "ant binary" is still not sufficient, since it makes a platform-specific tarball – the native code is only compiled for the architecture on which you're running, unless you pass some CXXFLAGS and CFLAGS environments.

          Plus, "ant binary" is useful for people who want to distribute a tarball around their cluster - it's the full "dist dir" minus the native stuff which some folks may not require.

          Show
          Todd Lipcon added a comment - For doing a full release binary tarball, "ant binary" is still not sufficient, since it makes a platform-specific tarball – the native code is only compiled for the architecture on which you're running, unless you pass some CXXFLAGS and CFLAGS environments. Plus, "ant binary" is useful for people who want to distribute a tarball around their cluster - it's the full "dist dir" minus the native stuff which some folks may not require.
          Hide
          Eric Yang added a comment -

          For secure version of hadoop, task-controller is essential. Secure hadoop should be the default for future version of hadoop, hence, I would vote for MAPREDUCE-2559-2.patch instead of MAPREDUCE-2559-3.patch.

          Show
          Eric Yang added a comment - For secure version of hadoop, task-controller is essential. Secure hadoop should be the default for future version of hadoop, hence, I would vote for MAPREDUCE-2559 -2.patch instead of MAPREDUCE-2559 -3.patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12481285/MAPREDUCE-2559-2.patch
          against trunk revision 1130554.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.cli.TestMRCLI

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/337//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/337//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/337//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481285/MAPREDUCE-2559-2.patch against trunk revision 1130554. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/337//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/337//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/337//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12481297/MAPREDUCE-2559-3.patch
          against trunk revision 1130554.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.cli.TestMRCLI

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/339//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/339//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/339//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481297/MAPREDUCE-2559-3.patch against trunk revision 1130554. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/339//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/339//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/339//console This message is automatically generated.
          Hide
          Eli Collins added a comment -

          I filed HDFS-2040 to make HDFS more consistent with Common and MR (where ant binary only builds native code if passed a flag). See that jira for rationale.

          Attaching mapreduce-2559-4.patch, minimal patch to allow ant binary to build the tarball as it did previously.

          Show
          Eli Collins added a comment - I filed HDFS-2040 to make HDFS more consistent with Common and MR (where ant binary only builds native code if passed a flag). See that jira for rationale. Attaching mapreduce-2559-4.patch, minimal patch to allow ant binary to build the tarball as it did previously.
          Eli Collins made changes -
          Attachment mapreduce-2559-4.patch [ 12481640 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12481640/mapreduce-2559-4.patch
          against trunk revision 1132807.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.cli.TestMRCLI
          org.apache.hadoop.fs.TestFileSystem

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/352//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/352//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/352//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481640/mapreduce-2559-4.patch against trunk revision 1132807. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI org.apache.hadoop.fs.TestFileSystem +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/352//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/352//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/352//console This message is automatically generated.
          Hide
          Eli Collins added a comment -

          I think the findbugs and test failures are unrelated to this patch.

          Eric, Todd - does this patch and the one on HDFS-2040 look reasonable to you guys?

          Show
          Eli Collins added a comment - I think the findbugs and test failures are unrelated to this patch. Eric, Todd - does this patch and the one on HDFS-2040 look reasonable to you guys?
          Hide
          Eric Yang added a comment -

          +1 on this patch, but HDFS-2040 patch may have problem.

          Show
          Eric Yang added a comment - +1 on this patch, but HDFS-2040 patch may have problem.
          Hide
          Todd Lipcon added a comment -

          +1 here.

          Show
          Todd Lipcon added a comment - +1 here.
          Hide
          Eli Collins added a comment -

          I've committed this. Thanks Eric and Todd.

          Show
          Eli Collins added a comment - I've committed this. Thanks Eric and Todd.
          Eli Collins made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #719 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/719/)
          MAPREDUCE-2559. ant binary fails due to missing c++ lib dir. Contributed by Eli Collins

          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1133226
          Files :

          • /hadoop/mapreduce/trunk/CHANGES.txt
          • /hadoop/mapreduce/trunk/build.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #719 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/719/ ) MAPREDUCE-2559 . ant binary fails due to missing c++ lib dir. Contributed by Eli Collins eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1133226 Files : /hadoop/mapreduce/trunk/CHANGES.txt /hadoop/mapreduce/trunk/build.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #722 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/722/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #722 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/722/ )
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Eric Yang
              Reporter:
              Eric Yang
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development