Hadoop Common
  1. Hadoop Common
  2. HADOOP-8480

The native build should honor -DskipTests

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Trivial Trivial
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.2-alpha
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently, the native build does not honor -DskipTests. The native unit tests will be run even when you specify:

      mvn compile -Pnative -DskipTests -X -e
      

      This seems inconsistent; shouldn't we fix this to work like the other tests do?

      1. HADOOP-8480.001.patch
        3 kB
        Colin Patrick McCabe
      2. HADOOP-8480.002.patch
        3 kB
        Colin Patrick McCabe

        Issue Links

          Activity

          Hide
          Colin Patrick McCabe added a comment -

          P.S. I'm working on a Maven plugin which will do the CMake / native tests stuff in a cleaner way. However, in the meantime, this patch should at least spare people the annoyance of running native tests when -DskipTests is specified.

          Show
          Colin Patrick McCabe added a comment - P.S. I'm working on a Maven plugin which will do the CMake / native tests stuff in a cleaner way. However, in the meantime, this patch should at least spare people the annoyance of running native tests when -DskipTests is specified.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12538607/HADOOP-8480.001.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:

          org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1241//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1241//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12538607/HADOOP-8480.001.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1241//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1241//console This message is automatically generated.
          Hide
          Alejandro Abdelnur added a comment -

          I've applied the patch and run mvn clean test -DskipTests -Pnative and HDFS native testcases still run. I've tested this in Ubuntu 10.4.

          Show
          Alejandro Abdelnur added a comment - I've applied the patch and run mvn clean test -DskipTests -Pnative and HDFS native testcases still run. I've tested this in Ubuntu 10.4.
          Hide
          Alejandro Abdelnur added a comment -

          After chatting with Colin offline I've tried changing the link /bin/sh from /bin/dash to /bin/bash and it works. Still we have to get this working in vanilla Ubuntu as we cannot assume we can tweak /bin/sh in the build machines.

          Show
          Alejandro Abdelnur added a comment - After chatting with Colin offline I've tried changing the link /bin/sh from /bin/dash to /bin/bash and it works. Still we have to get this working in vanilla Ubuntu as we cannot assume we can tweak /bin/sh in the build machines.
          Hide
          Colin Patrick McCabe added a comment -
          • don't use double equals sign (it only works on bash, not dash). Single equals sign is actually the right thing to use here in both bash and dash.
          Show
          Colin Patrick McCabe added a comment - don't use double equals sign (it only works on bash, not dash). Single equals sign is actually the right thing to use here in both bash and dash.
          Hide
          Andy Isaacson added a comment -

          That's a pretty awful hack, but there's no better way AFAIK. LGTM. Patch .002 should work for all shells.

          It does break for certain evil values, if the user specifies -DskipTests="foo bar" I think the shell expansion will throw an error and we will run the tests. Which is, sort of, what the user asked for, so I think it's OK.

          Show
          Andy Isaacson added a comment - That's a pretty awful hack, but there's no better way AFAIK. LGTM. Patch .002 should work for all shells. It does break for certain evil values, if the user specifies -DskipTests="foo bar" I think the shell expansion will throw an error and we will run the tests. Which is, sort of, what the user asked for, so I think it's OK.
          Hide
          Alejandro Abdelnur added a comment -

          +1, tested in ubuntu with the default sh->dash and it works as expected. regarding Andy's comment on -DskipTests="foo bar", not sure that will every happen, so I wouldn't worry about it.

          Show
          Alejandro Abdelnur added a comment - +1, tested in ubuntu with the default sh->dash and it works as expected. regarding Andy's comment on -DskipTests="foo bar", not sure that will every happen, so I wouldn't worry about it.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12538812/HADOOP-8480.002.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:

          org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1244//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1244//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12538812/HADOOP-8480.002.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1244//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1244//console This message is automatically generated.
          Hide
          Eli Collins added a comment -

          Test failure is unrelated. I've committed this and merged to branch-2.

          Show
          Eli Collins added a comment - Test failure is unrelated. I've committed this and merged to branch-2.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2613 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2613/)
          HADOOP-8480. The native build should honor -DskipTests. Contributed by Colin Patrick McCabe (Revision 1368257)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1368257
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2613 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2613/ ) HADOOP-8480 . The native build should honor -DskipTests. Contributed by Colin Patrick McCabe (Revision 1368257) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1368257 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2548 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2548/)
          HADOOP-8480. The native build should honor -DskipTests. Contributed by Colin Patrick McCabe (Revision 1368257)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1368257
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2548 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2548/ ) HADOOP-8480 . The native build should honor -DskipTests. Contributed by Colin Patrick McCabe (Revision 1368257) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1368257 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2566 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2566/)
          HADOOP-8480. The native build should honor -DskipTests. Contributed by Colin Patrick McCabe (Revision 1368257)

          Result = FAILURE
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1368257
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2566 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2566/ ) HADOOP-8480 . The native build should honor -DskipTests. Contributed by Colin Patrick McCabe (Revision 1368257) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1368257 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1123 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1123/)
          HADOOP-8480. The native build should honor -DskipTests. Contributed by Colin Patrick McCabe (Revision 1368257)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1368257
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1123 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1123/ ) HADOOP-8480 . The native build should honor -DskipTests. Contributed by Colin Patrick McCabe (Revision 1368257) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1368257 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1155 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1155/)
          HADOOP-8480. The native build should honor -DskipTests. Contributed by Colin Patrick McCabe (Revision 1368257)

          Result = FAILURE
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1368257
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1155 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1155/ ) HADOOP-8480 . The native build should honor -DskipTests. Contributed by Colin Patrick McCabe (Revision 1368257) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1368257 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml
          Hide
          Jiahongchao added a comment -

          I tried to build native library for hadoop 2.3 in 64bit Centos, this issus happed again.
          mvn package -Pdist,native -Dskiptests -Dtar always failed .

          Show
          Jiahongchao added a comment - I tried to build native library for hadoop 2.3 in 64bit Centos, this issus happed again. mvn package -Pdist,native -Dskiptests -Dtar always failed .
          Hide
          Colin Patrick McCabe added a comment -

          mvn package -Pdist,native -Dskiptests -Dtar always failed .

          What failed and how?

          Show
          Colin Patrick McCabe added a comment - mvn package -Pdist,native -Dskiptests -Dtar always failed . What failed and how?
          Hide
          Jiahongchao added a comment -

          I ran
          mvn package -Pdist,native -Dskiptests -Dtar
          for libhadoop.so file. That command failed when running some tests. I forgot which testcase.Finally I ran mvn compile instead

          Show
          Jiahongchao added a comment - I ran mvn package -Pdist,native -Dskiptests -Dtar for libhadoop.so file. That command failed when running some tests. I forgot which testcase.Finally I ran mvn compile instead
          Hide
          Sam Clus added a comment -

          I have solved this problem already! Here is my solution:
          1. find the file : hadoop-hdfs-project/pom.xml
          2. open the file,search "skip" , find the label , change the value "false" to "true".

          If you have any problem , you can ask me : QQ international:940417022.

          Show
          Sam Clus added a comment - I have solved this problem already! Here is my solution: 1. find the file : hadoop-hdfs-project/pom.xml 2. open the file,search "skip" , find the label , change the value "false" to "true". If you have any problem , you can ask me : QQ international:940417022.

            People

            • Assignee:
              Colin Patrick McCabe
              Reporter:
              Colin Patrick McCabe
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development