Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.22.0, 0.23.0
    • Component/s: build
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      The native build run when from trunk now requires autotools, libtool and openssl dev libraries.

      Description

      The repo currently includes the automake and autoconf generated files for the native build. Per discussion on HADOOP-6421 let's remove them and use the host's automake and autoconf. We should also do this for libhdfs and fuse-dfs.

      1. MAPREDUCE-2260.diff
        3.94 MB
        Roman Shaposhnik

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-22-branch #33 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/33/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-22-branch #33 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/33/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #595 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/595/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #595 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/595/ )
          Hide
          Eli Collins added a comment -

          I've committed this.

          Show
          Eli Collins added a comment - I've committed this.
          Hide
          Eli Collins added a comment - - edited

          +1

          I verified ant -Dcompile.native=true -Dcompile.c+=yes compile-c+-examples task-controller builds correctly on 64-bit Centos 5 and 64-bit Ubuntu Maverick. And that the following commands can be used to regenerate the configure files:

          ant create-c++-utils-configure -Dneed.c++.utils.configure=true
          ant create-c++-pipes-configure need.c++.pipes.configure
          ant create-c++-examples-pipes-configure need.c++.examples.pipes.configure
          ant create-c++-task-controller-configure need.c++.task-controller.configure
          

          I will commit this to trunk and branch 22. Thanks Roman!

          Show
          Eli Collins added a comment - - edited +1 I verified ant -Dcompile.native=true -Dcompile.c+ =yes compile-c +-examples task-controller builds correctly on 64-bit Centos 5 and 64-bit Ubuntu Maverick. And that the following commands can be used to regenerate the configure files: ant create-c++-utils-configure -Dneed.c++.utils.configure=true ant create-c++-pipes-configure need.c++.pipes.configure ant create-c++-examples-pipes-configure need.c++.examples.pipes.configure ant create-c++-task-controller-configure need.c++.task-controller.configure I will commit this to trunk and branch 22. Thanks Roman!
          Hide
          Roman Shaposhnik added a comment -

          To finish up my previous answer – the command used to build hadoop-mapreduce was:

          ant -Dcompile.native=true -Dcompile.c+=true -Djava5.home=$JAVA5_HOME -Dforrest.home=$FORREST_HOME -Dhadoop.conf.dir=/etc/hadoop-0.20/conf -Dlibrecordio=true veryclean api-report task-controller compile-c+ tar

          Finally, as for testing – examples (pipes) were executed

          Show
          Roman Shaposhnik added a comment - To finish up my previous answer – the command used to build hadoop-mapreduce was: ant -Dcompile.native=true -Dcompile.c+ =true -Djava5.home=$JAVA5_HOME -Dforrest.home=$FORREST_HOME -Dhadoop.conf.dir=/etc/hadoop-0.20/conf -Dlibrecordio=true veryclean api-report task-controller compile-c + tar Finally, as for testing – examples (pipes) were executed
          Hide
          Roman Shaposhnik added a comment -

          it would be good to verify that ... ant tar produces a tarball which contains the generated files (ie a dist style tarball).

          Since one has to specify extra properties in order for native bits to be included in a dist style tarball here's a list of builds that were done:

          cd hadoop-common ; ant -Dcompile.native=true -Dcompile.c++=true -Djava5.home=$JAVA5_HOME -Dforrest.home=$FORREST_HOME -Dhadoop.conf.dir=/etc/hadoop/conf veryclean api-report compile-core-native tar
          cd hadoop-hdfs ; ant -Dcompile.native=true -Dcompile.c++=true -Djava5.home=$JAVA5_HOME -Dforrest.home=$FORREST_HOME -Dhadoop.conf.dir=/etc/hadoop/conf -Dlibhdfs=1 veryclean api-report tar
          cd hadoop-mapreduce ;

          That resulted in the following native bits being created:

          c++/Linux-amd64-64/include/hadoop/Pipes.hh
          c++/Linux-amd64-64/include/hadoop/SerialUtils.hh
          c++/Linux-amd64-64/include/hadoop/TemplateFactory.hh
          c++/Linux-amd64-64/include/hadoop/StringUtils.hh
          c++/Linux-amd64-64/lib/libhadooppipes.a
          c++/Linux-amd64-64/lib/libhadooputils.a
          c++/Linux-amd64-64/lib/libhdfs.a
          c++/Linux-amd64-64/lib/libhdfs.so.0.0.0
          c++/Linux-amd64-64/lib/libhdfs.so.0
          c++/Linux-amd64-64/lib/libhdfs.so
          c++/Linux-amd64-64/lib/libhdfs.la
          c++/lib/libhdfs.a
          c++/lib/libhdfs.so.0.0.0
          c++/lib/libhdfs.so.0
          c++/lib/libhdfs.so
          c++/lib/libhdfs.la
          librecordio/librecordio.a
          bin/task-controller
          lib/native/Linux-amd64-64/libhadoop.so
          lib/native/Linux-amd64-64/libhadoop.a
          lib/native/Linux-amd64-64/libhadoop.so.1
          lib/native/Linux-amd64-64/libhadoop.so.1.0.0
          lib/native/Linux-amd64-64/libhadoop.la

          Show
          Roman Shaposhnik added a comment - it would be good to verify that ... ant tar produces a tarball which contains the generated files (ie a dist style tarball). Since one has to specify extra properties in order for native bits to be included in a dist style tarball here's a list of builds that were done: cd hadoop-common ; ant -Dcompile.native=true -Dcompile.c++=true -Djava5.home=$JAVA5_HOME -Dforrest.home=$FORREST_HOME -Dhadoop.conf.dir=/etc/hadoop/conf veryclean api-report compile-core-native tar cd hadoop-hdfs ; ant -Dcompile.native=true -Dcompile.c++=true -Djava5.home=$JAVA5_HOME -Dforrest.home=$FORREST_HOME -Dhadoop.conf.dir=/etc/hadoop/conf -Dlibhdfs=1 veryclean api-report tar cd hadoop-mapreduce ; That resulted in the following native bits being created: c++/Linux-amd64-64/include/hadoop/Pipes.hh c++/Linux-amd64-64/include/hadoop/SerialUtils.hh c++/Linux-amd64-64/include/hadoop/TemplateFactory.hh c++/Linux-amd64-64/include/hadoop/StringUtils.hh c++/Linux-amd64-64/lib/libhadooppipes.a c++/Linux-amd64-64/lib/libhadooputils.a c++/Linux-amd64-64/lib/libhdfs.a c++/Linux-amd64-64/lib/libhdfs.so.0.0.0 c++/Linux-amd64-64/lib/libhdfs.so.0 c++/Linux-amd64-64/lib/libhdfs.so c++/Linux-amd64-64/lib/libhdfs.la c++/lib/libhdfs.a c++/lib/libhdfs.so.0.0.0 c++/lib/libhdfs.so.0 c++/lib/libhdfs.so c++/lib/libhdfs.la librecordio/librecordio.a bin/task-controller lib/native/Linux-amd64-64/libhadoop.so lib/native/Linux-amd64-64/libhadoop.a lib/native/Linux-amd64-64/libhadoop.so.1 lib/native/Linux-amd64-64/libhadoop.so.1.0.0 lib/native/Linux-amd64-64/libhadoop.la
          Hide
          Eli Collins added a comment -

          Ignore my previous comment, there was a bug in the tool I was using to view the diffs. The latest patch looks great.

          Per HADOOP-6436 please update with what testing you've done. I checked the binaries themselves (eg just metadata changes) so I don't think this breaks anything but it would be good to verify that pipes, libhdfs, etc still run and that ant tar produces a tarball which contains the generated files (ie a dist style tarball).

          It would be nice if we used VPATH builds so the source directory wasn't cluttered with autoconf files but that's orthogonal to this change.

          Thanks,
          Eli

          Show
          Eli Collins added a comment - Ignore my previous comment, there was a bug in the tool I was using to view the diffs. The latest patch looks great. Per HADOOP-6436 please update with what testing you've done. I checked the binaries themselves (eg just metadata changes) so I don't think this breaks anything but it would be good to verify that pipes, libhdfs, etc still run and that ant tar produces a tarball which contains the generated files (ie a dist style tarball). It would be nice if we used VPATH builds so the source directory wasn't cluttered with autoconf files but that's orthogonal to this change. Thanks, Eli
          Hide
          Eli Collins added a comment -

          Overall the structure looks great. The configure.ac for taskcontroller and utils doesn't need to reference USE_HADOOP_PIPES, wordcount-simple.cc, and hadoop-pipes-examples. Otherwise looks good.

          Show
          Eli Collins added a comment - Overall the structure looks great. The configure.ac for taskcontroller and utils doesn't need to reference USE_HADOOP_PIPES, wordcount-simple.cc, and hadoop-pipes-examples. Otherwise looks good.
          Hide
          Roman Shaposhnik added a comment -

          I'm attaching a first cut at the patch. Please let me know what do you think about the approach taken.

          Notes:

          1. since none of the C++ projects is imported (they are all developed as part of the Hadoop mapreduce workspace)
          it seems that we all would be better off defining a single common umbrella C++ project with a single autotools
          metastructure and 5 different subdirectories corresponding to pipes, pipes-examples, task-controller, utils and librecordio.
          We don't really gain anything by splitting C++ sources at the level of Ant.

          2. the build (at least the release build) now depends on the entire autoconf

          Show
          Roman Shaposhnik added a comment - I'm attaching a first cut at the patch. Please let me know what do you think about the approach taken. Notes: 1. since none of the C++ projects is imported (they are all developed as part of the Hadoop mapreduce workspace) it seems that we all would be better off defining a single common umbrella C++ project with a single autotools metastructure and 5 different subdirectories corresponding to pipes, pipes-examples, task-controller, utils and librecordio. We don't really gain anything by splitting C++ sources at the level of Ant. 2. the build (at least the release build) now depends on the entire autoconf

            People

            • Assignee:
              Roman Shaposhnik
              Reporter:
              Roman Shaposhnik
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 24h
                24h
                Remaining:
                Remaining Estimate - 24h
                24h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development