Hadoop Common
  1. Hadoop Common
  2. HADOOP-8620

Add -Drequire.fuse and -Drequire.snappy

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.0.1-alpha
    • Fix Version/s: 2.0.2-alpha
    • Component/s: build
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We have some optional build components which don't get built if they are not installed on the build machine. One of those components is fuse_dfs. Another is the snappy support in libhadoop.so.

      Unfortunately, since these components are silently ignored if they are not present, it's easy to unintentionally create an incomplete build.

      We should add two flags, -Drequire.fuse and -Drequire.snappy, that do exactly what the names suggest. This will make the build more repeatable. Those who want a complete build can specify these system properties to maven. If the build cannot be created as requested, it will be a hard error.

      1. HADOOP-8620.002.patch
        4 kB
        Colin Patrick McCabe
      2. HADOOP-8620.003.patch
        6 kB
        Colin Patrick McCabe

        Issue Links

          Activity

          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12537762/HADOOP-8620.002.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.ha.TestZKFailoverController
          org.apache.hadoop.hdfs.TestDFSClientRetries
          org.apache.hadoop.hdfs.TestPersistBlocks

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1216//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1216//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12537762/HADOOP-8620.002.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.ha.TestZKFailoverController org.apache.hadoop.hdfs.TestDFSClientRetries org.apache.hadoop.hdfs.TestPersistBlocks +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1216//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1216//console This message is automatically generated.
          Hide
          Colin Patrick McCabe added a comment -

          The test failures are not related to the patch-- this is purely a build system thing.

          Show
          Colin Patrick McCabe added a comment - The test failures are not related to the patch-- this is purely a build system thing.
          Hide
          Alejandro Abdelnur added a comment -

          I've tried the mvn clean test -DskipTests -Drequire.snappy=true -Dsnappy.prefix=/home/tucu/src/snappy-1.0.3/build/usr/local -Pnative on Ubuntu 10.4 and the build is failing and I have snappy in the specified directory:

          /home/tucu/src/snappy-1.0.3/build/usr/local/
          /home/tucu/src/snappy-1.0.3/build/usr/local/share
          /home/tucu/src/snappy-1.0.3/build/usr/local/share/doc
          /home/tucu/src/snappy-1.0.3/build/usr/local/share/doc/snappy
          /home/tucu/src/snappy-1.0.3/build/usr/local/share/doc/snappy/INSTALL
          /home/tucu/src/snappy-1.0.3/build/usr/local/share/doc/snappy/ChangeLog
          /home/tucu/src/snappy-1.0.3/build/usr/local/share/doc/snappy/COPYING
          /home/tucu/src/snappy-1.0.3/build/usr/local/share/doc/snappy/README
          /home/tucu/src/snappy-1.0.3/build/usr/local/share/doc/snappy/format_description.txt
          /home/tucu/src/snappy-1.0.3/build/usr/local/share/doc/snappy/NEWS
          /home/tucu/src/snappy-1.0.3/build/usr/local/include
          /home/tucu/src/snappy-1.0.3/build/usr/local/include/snappy.h
          /home/tucu/src/snappy-1.0.3/build/usr/local/include/snappy-stubs-public.h
          /home/tucu/src/snappy-1.0.3/build/usr/local/include/snappy-c.h
          /home/tucu/src/snappy-1.0.3/build/usr/local/include/snappy-sinksource.h
          /home/tucu/src/snappy-1.0.3/build/usr/local/lib
          /home/tucu/src/snappy-1.0.3/build/usr/local/lib/libsnappy.a
          /home/tucu/src/snappy-1.0.3/build/usr/local/lib/libsnappy.so
          /home/tucu/src/snappy-1.0.3/build/usr/local/lib/libsnappy.so.1
          /home/tucu/src/snappy-1.0.3/build/usr/local/lib/libsnappy.la
          /home/tucu/src/snappy-1.0.3/build/usr/local/lib/libsnappy.so.1.1.1
          
          Show
          Alejandro Abdelnur added a comment - I've tried the mvn clean test -DskipTests -Drequire.snappy=true -Dsnappy.prefix=/home/tucu/src/snappy-1.0.3/build/usr/local -Pnative on Ubuntu 10.4 and the build is failing and I have snappy in the specified directory: /home/tucu/src/snappy-1.0.3/build/usr/local/ /home/tucu/src/snappy-1.0.3/build/usr/local/share /home/tucu/src/snappy-1.0.3/build/usr/local/share/doc /home/tucu/src/snappy-1.0.3/build/usr/local/share/doc/snappy /home/tucu/src/snappy-1.0.3/build/usr/local/share/doc/snappy/INSTALL /home/tucu/src/snappy-1.0.3/build/usr/local/share/doc/snappy/ChangeLog /home/tucu/src/snappy-1.0.3/build/usr/local/share/doc/snappy/COPYING /home/tucu/src/snappy-1.0.3/build/usr/local/share/doc/snappy/README /home/tucu/src/snappy-1.0.3/build/usr/local/share/doc/snappy/format_description.txt /home/tucu/src/snappy-1.0.3/build/usr/local/share/doc/snappy/NEWS /home/tucu/src/snappy-1.0.3/build/usr/local/include /home/tucu/src/snappy-1.0.3/build/usr/local/include/snappy.h /home/tucu/src/snappy-1.0.3/build/usr/local/include/snappy-stubs- public .h /home/tucu/src/snappy-1.0.3/build/usr/local/include/snappy-c.h /home/tucu/src/snappy-1.0.3/build/usr/local/include/snappy-sinksource.h /home/tucu/src/snappy-1.0.3/build/usr/local/lib /home/tucu/src/snappy-1.0.3/build/usr/local/lib/libsnappy.a /home/tucu/src/snappy-1.0.3/build/usr/local/lib/libsnappy.so /home/tucu/src/snappy-1.0.3/build/usr/local/lib/libsnappy.so.1 /home/tucu/src/snappy-1.0.3/build/usr/local/lib/libsnappy.la /home/tucu/src/snappy-1.0.3/build/usr/local/lib/libsnappy.so.1.1.1
          Hide
          Colin Patrick McCabe added a comment -
          • remove -Dbundle.snappy because we don't support it. This has been obsolete for a long, long time.
          • remove runas.home. runas was removed by HADOOP-8450, but we forgot to remove runas.home from the pom.xml file.
          • fix -Dsnappy.prefix, -Dsnappy.lib, -Dsnappy.include. They were broken by HADOOP-8368.
          Show
          Colin Patrick McCabe added a comment - remove -Dbundle.snappy because we don't support it. This has been obsolete for a long, long time. remove runas.home. runas was removed by HADOOP-8450 , but we forgot to remove runas.home from the pom.xml file. fix -Dsnappy.prefix, -Dsnappy.lib, -Dsnappy.include. They were broken by HADOOP-8368 .
          Hide
          Alejandro Abdelnur added a comment -

          +1. run build requiring with/without snappy, works as advertised.

          Show
          Alejandro Abdelnur added a comment - +1. run build requiring with/without snappy, works as advertised.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12538811/HADOOP-8620.003.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.ha.TestZKFailoverController
          org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl
          org.apache.hadoop.hdfs.server.datanode.TestBlockReport
          org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1243//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1243//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12538811/HADOOP-8620.003.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.ha.TestZKFailoverController org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl org.apache.hadoop.hdfs.server.datanode.TestBlockReport org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1243//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1243//console This message is automatically generated.
          Hide
          Eli Collins added a comment -

          Test failures are unrelated. I've committed this and merged to branch-2. Thanks Colin.

          Show
          Eli Collins added a comment - Test failures are unrelated. I've committed this and merged to branch-2. Thanks Colin.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2613 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2613/)
          HADOOP-8620. Add -Drequire.fuse and -Drequire.snappy. Contributed by Colin Patrick McCabe (Revision 1368251)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1368251
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/CMakeLists.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2613 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2613/ ) HADOOP-8620 . Add -Drequire.fuse and -Drequire.snappy. Contributed by Colin Patrick McCabe (Revision 1368251) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1368251 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/CMakeLists.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2548 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2548/)
          HADOOP-8620. Add -Drequire.fuse and -Drequire.snappy. Contributed by Colin Patrick McCabe (Revision 1368251)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1368251
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/CMakeLists.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2548 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2548/ ) HADOOP-8620 . Add -Drequire.fuse and -Drequire.snappy. Contributed by Colin Patrick McCabe (Revision 1368251) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1368251 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/CMakeLists.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2566 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2566/)
          HADOOP-8620. Add -Drequire.fuse and -Drequire.snappy. Contributed by Colin Patrick McCabe (Revision 1368251)

          Result = FAILURE
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1368251
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/CMakeLists.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2566 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2566/ ) HADOOP-8620 . Add -Drequire.fuse and -Drequire.snappy. Contributed by Colin Patrick McCabe (Revision 1368251) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1368251 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/CMakeLists.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1123 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1123/)
          HADOOP-8620. Add -Drequire.fuse and -Drequire.snappy. Contributed by Colin Patrick McCabe (Revision 1368251)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1368251
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/CMakeLists.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1123 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1123/ ) HADOOP-8620 . Add -Drequire.fuse and -Drequire.snappy. Contributed by Colin Patrick McCabe (Revision 1368251) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1368251 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/CMakeLists.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1155 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1155/)
          HADOOP-8620. Add -Drequire.fuse and -Drequire.snappy. Contributed by Colin Patrick McCabe (Revision 1368251)

          Result = FAILURE
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1368251
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/CMakeLists.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1155 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1155/ ) HADOOP-8620 . Add -Drequire.fuse and -Drequire.snappy. Contributed by Colin Patrick McCabe (Revision 1368251) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1368251 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/CMakeLists.txt

            People

            • Assignee:
              Colin Patrick McCabe
              Reporter:
              Colin Patrick McCabe
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development