Hadoop Common
  1. Hadoop Common
  2. HADOOP-8547

Package hadoop-pipes examples/bin directory (again)

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.2-alpha
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      It looks like since MAPREDUCE-4267, we're no longer exporting the hadooppipes examples/bin directory to hadoop-dist as part of a "mvn package" build. This seems unintentional, so we should export those binaries again.

      1. HDFS-3589.001.patch
        2 kB
        Colin Patrick McCabe

        Issue Links

          Activity

          Hide
          Eli Collins added a comment -

          Why remove the installation of the headers? Otherwise looks good.

          Show
          Eli Collins added a comment - Why remove the installation of the headers? Otherwise looks good.
          Hide
          Colin Patrick McCabe added a comment -

          The headers don't need to be installed because hadoop-tools.xml takes them directly out of the source tree.

              <fileSet>
                <directory>../hadoop-pipes/src/main/native/utils/api/hadoop</directory>
                <includes>
                  <include>*.hh</include>
                </includes>
                <outputDirectory>/include</outputDirectory>
              </fileSet>
          
          cmccabe@keter:~/hadoop5> ls -l  hadoop-tools/hadoop-pipes/src/main/native/utils/api/hadoop/
          total 12
          -rw-r--r-- 1 cmccabe users 4514 Jul  2 10:34 SerialUtils.hh
          -rw-r--r-- 1 cmccabe users 2441 Jul  2 10:34 StringUtils.hh
          

          Another way to see that this is true is to realize that we never call "make install" so all INSTALL sections are null and void anyway.

          Show
          Colin Patrick McCabe added a comment - The headers don't need to be installed because hadoop-tools.xml takes them directly out of the source tree. <fileSet> <directory>../hadoop-pipes/src/main/ native /utils/api/hadoop</directory> <includes> <include>*.hh</include> </includes> <outputDirectory>/include</outputDirectory> </fileSet> cmccabe@keter:~/hadoop5> ls -l hadoop-tools/hadoop-pipes/src/main/ native /utils/api/hadoop/ total 12 -rw-r--r-- 1 cmccabe users 4514 Jul 2 10:34 SerialUtils.hh -rw-r--r-- 1 cmccabe users 2441 Jul 2 10:34 StringUtils.hh Another way to see that this is true is to realize that we never call "make install" so all INSTALL sections are null and void anyway.
          Hide
          Eli Collins added a comment -

          Ah, great. +1 pending jenkins

          Show
          Eli Collins added a comment - Ah, great. +1 pending jenkins
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12534296/HDFS-3589.001.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-assemblies.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1163//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1163//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12534296/HDFS-3589.001.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-assemblies. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1163//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1163//console This message is automatically generated.
          Hide
          Colin Patrick McCabe added a comment -

          patch doesn't require new tests because it's not a code change, just a packaging change

          Show
          Colin Patrick McCabe added a comment - patch doesn't require new tests because it's not a code change, just a packaging change
          Hide
          Eli Collins added a comment -

          I've committed this to trunk and merged to branch-2. Thanks Colin!

          Show
          Eli Collins added a comment - I've committed this to trunk and merged to branch-2. Thanks Colin!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2422 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2422/)
          HADOOP-8547. Package hadoop-pipes examples/bin directory (again). Contributed by Colin Patrick McCabe (Revision 1356947)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1356947
          Files :

          • /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/CMakeLists.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2422 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2422/ ) HADOOP-8547 . Package hadoop-pipes examples/bin directory (again). Contributed by Colin Patrick McCabe (Revision 1356947) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1356947 Files : /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/CMakeLists.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2490 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2490/)
          HADOOP-8547. Package hadoop-pipes examples/bin directory (again). Contributed by Colin Patrick McCabe (Revision 1356947)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1356947
          Files :

          • /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/CMakeLists.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2490 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2490/ ) HADOOP-8547 . Package hadoop-pipes examples/bin directory (again). Contributed by Colin Patrick McCabe (Revision 1356947) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1356947 Files : /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/CMakeLists.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2439 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2439/)
          HADOOP-8547. Package hadoop-pipes examples/bin directory (again). Contributed by Colin Patrick McCabe (Revision 1356947)

          Result = FAILURE
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1356947
          Files :

          • /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/CMakeLists.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2439 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2439/ ) HADOOP-8547 . Package hadoop-pipes examples/bin directory (again). Contributed by Colin Patrick McCabe (Revision 1356947) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1356947 Files : /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/CMakeLists.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1094 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1094/)
          HADOOP-8547. Package hadoop-pipes examples/bin directory (again). Contributed by Colin Patrick McCabe (Revision 1356947)

          Result = FAILURE
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1356947
          Files :

          • /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/CMakeLists.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1094 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1094/ ) HADOOP-8547 . Package hadoop-pipes examples/bin directory (again). Contributed by Colin Patrick McCabe (Revision 1356947) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1356947 Files : /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/CMakeLists.txt
          Hide
          Thomas Graves added a comment -

          Sorry for coming in late on this, jira has been so flaky lately haven't kept up.

          I had actually intentionally left those out. They weren't in the 1.X dist package and aren't needed to actually build a pipes program. I thought the community sentiment was to move away from pipes also so was just included what was essential in the dist package. They also aren't very well documented so I think that should be fixed if we are going to include them in the distribution. When I just look at examples bin it isn't obvious to me how to run them or what they are for (pipes).

          Show
          Thomas Graves added a comment - Sorry for coming in late on this, jira has been so flaky lately haven't kept up. I had actually intentionally left those out. They weren't in the 1.X dist package and aren't needed to actually build a pipes program. I thought the community sentiment was to move away from pipes also so was just included what was essential in the dist package. They also aren't very well documented so I think that should be fixed if we are going to include them in the distribution. When I just look at examples bin it isn't obvious to me how to run them or what they are for (pipes).
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1127 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1127/)
          HADOOP-8547. Package hadoop-pipes examples/bin directory (again). Contributed by Colin Patrick McCabe (Revision 1356947)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1356947
          Files :

          • /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/CMakeLists.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1127 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1127/ ) HADOOP-8547 . Package hadoop-pipes examples/bin directory (again). Contributed by Colin Patrick McCabe (Revision 1356947) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1356947 Files : /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/CMakeLists.txt
          Hide
          Eli Collins added a comment -

          @Thomas, thanks for chiming in. How about a jira to remove the pipes examples entirely (or docs on how to run them, what they're for etc)?

          Show
          Eli Collins added a comment - @Thomas, thanks for chiming in. How about a jira to remove the pipes examples entirely (or docs on how to run them, what they're for etc)?
          Hide
          Thomas Graves added a comment -

          I don't think we need to remove them entirely because they were useful to me to test pipes when I mavenized it. I think documenting would be good, there is a README.txt down in the examples directory itself. We could also add some to https://issues.apache.org/jira/browse/HADOOP-8477 about the examples.

          If someone finds them useful in the distribution I'm fine with them being there although would personally prefer to see them in a pipes subdir (examples/pipes/bin) or renamed to have pipes in the name or something so that its obvious what they are for. If there isn't a good use case, I would prefer to see this reverted as in my opinion it just adds more clutter to the distribution. Personally if I'm writing a pipes program and want an example, I'm going to want both the source code and the binary so I don't see the binaries by themselves very useful.

          Show
          Thomas Graves added a comment - I don't think we need to remove them entirely because they were useful to me to test pipes when I mavenized it. I think documenting would be good, there is a README.txt down in the examples directory itself. We could also add some to https://issues.apache.org/jira/browse/HADOOP-8477 about the examples. If someone finds them useful in the distribution I'm fine with them being there although would personally prefer to see them in a pipes subdir (examples/pipes/bin) or renamed to have pipes in the name or something so that its obvious what they are for. If there isn't a good use case, I would prefer to see this reverted as in my opinion it just adds more clutter to the distribution. Personally if I'm writing a pipes program and want an example, I'm going to want both the source code and the binary so I don't see the binaries by themselves very useful.
          Hide
          Colin Patrick McCabe added a comment -

          There was a small amount of cleanup to the CMakeLists.txt that I did in the patch for HADOOP-8547 so I'd rather not do a straight up revert.

          I posted a patch to HADOOP-8563 that removes the examples/bin from the packaging directory; check it out when you get a chance.

          Show
          Colin Patrick McCabe added a comment - There was a small amount of cleanup to the CMakeLists.txt that I did in the patch for HADOOP-8547 so I'd rather not do a straight up revert. I posted a patch to HADOOP-8563 that removes the examples/bin from the packaging directory; check it out when you get a chance.

            People

            • Assignee:
              Colin Patrick McCabe
              Reporter:
              Colin Patrick McCabe
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development