Uploaded image for project: 'Maven Surefire'
  1. Maven Surefire
  2. SUREFIRE-827

Surefire 2.12 cannot run a single test, regression from 2.11

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.12
    • Fix Version/s: 2.12.1
    • Labels:
      None
    • Environment:
      Ubuntu 11.10

      Description

      1. Surefire 2.11
        $ mvn test -Dtest=DataTest#testDataServerGetNonExistentFile
        ...
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
      1. Surefire 2.12
        mvn test -Dtest=DataTest#testDataServerGetNonExistentFile
        ...
        Tests run: 9, Failures: 0, Errors: 0, Skipped: 0
      1. BUG-827.zip
        2 kB
        Falko Modler

        Issue Links

          Activity

          Hide
          famod Falko Modler added a comment - - edited

          Similar problem here:

          -Dtest=SomeTest does not run any test an raises an error:

          ...
          Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
          
          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD FAILURE
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 4:20.351s
          [INFO] Finished at: Thu Feb 02 23:58:33 CET 2012
          [INFO] Final Memory: 17M/981M
          [INFO] ------------------------------------------------------------------------
          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12:test (default-test) on project someproject: No tests were executed!  (Set -DfailIfNoTests=false to ignore this error.) -> [Help 1]
          

          I tried to debug the problem bu got lost in ForkStarter.java which received a lot of changes since 2.11.

          Env: Java6 u29, Maven 3.0.4.

          Show
          famod Falko Modler added a comment - - edited Similar problem here: -Dtest=SomeTest does not run any test an raises an error: ... Tests run: 0, Failures: 0, Errors: 0, Skipped: 0 [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 4:20.351s [INFO] Finished at: Thu Feb 02 23:58:33 CET 2012 [INFO] Final Memory: 17M/981M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12:test ( default -test) on project someproject: No tests were executed! (Set -DfailIfNoTests= false to ignore this error.) -> [Help 1] I tried to debug the problem bu got lost in ForkStarter.java which received a lot of changes since 2.11. Env: Java6 u29, Maven 3.0.4.
          Hide
          olamy Olivier Lamy (*$^¨%`£) added a comment -

          junit or testng ?

          Show
          olamy Olivier Lamy (*$^¨%`£) added a comment - junit or testng ?
          Hide
          famod Falko Modler added a comment -

          Junit.

          Show
          famod Falko Modler added a comment - Junit.
          Hide
          h2o Holger Hoffstätte added a comment -

          Confirmed with maven 3.0.4/surefire 2.12/junit-4.10. Switching to 2.11 immediately works.

          Show
          h2o Holger Hoffstätte added a comment - Confirmed with maven 3.0.4/surefire 2.12/junit-4.10. Switching to 2.11 immediately works.
          Hide
          olamy Olivier Lamy (*$^¨%`£) added a comment -

          I have tested a project with junit 4.8.2 and this feature works.
          Do you have any sample project to reproduce ?

          Show
          olamy Olivier Lamy (*$^¨%`£) added a comment - I have tested a project with junit 4.8.2 and this feature works. Do you have any sample project to reproduce ?
          Hide
          gaul Andrew Gaul added a comment - - edited

          Oliver I believe that this is related to parallel tests and is a regression from SUREFIRE-760. Given an Apache commons-lang 3.1 project and setting the surefire version in the pom to $

          {surefire.version}

          , I get these results:

          $ mvn test -Dtest=ImmutablePairTest#testBasic -Dparallel=classes -Dsurefire.version=2.11
          ...
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
          
          $ mvn test -Dtest=ImmutablePairTest#testBasic -Dparallel=classes -Dsurefire.version=2.12
          ...
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0
          
          $ mvn test -Dtest=ImmutablePairTest#testBasic -Dparallel=none -Dsurefire.version=2.11
          ...
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
          
          $ mvn test -Dtest=ImmutablePairTest#testBasic -Dparallel=none -Dsurefire.version=2.12
          ...
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
          

          I cannot speak to Falko's symptoms.

          Show
          gaul Andrew Gaul added a comment - - edited Oliver I believe that this is related to parallel tests and is a regression from SUREFIRE-760 . Given an Apache commons-lang 3.1 project and setting the surefire version in the pom to $ {surefire.version} , I get these results: $ mvn test -Dtest=ImmutablePairTest#testBasic -Dparallel=classes -Dsurefire.version=2.11 ... Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 $ mvn test -Dtest=ImmutablePairTest#testBasic -Dparallel=classes -Dsurefire.version=2.12 ... Tests run: 6, Failures: 0, Errors: 0, Skipped: 0 $ mvn test -Dtest=ImmutablePairTest#testBasic -Dparallel=none -Dsurefire.version=2.11 ... Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 $ mvn test -Dtest=ImmutablePairTest#testBasic -Dparallel=none -Dsurefire.version=2.12 ... Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 I cannot speak to Falko's symptoms.
          Hide
          fleiber François LEIBER added a comment - - edited

          We also have a big issue with surefire 2.12 (using default configuration, which means forkMode=once): the Junit tests of the first module pass perfectly, then we have a "Z,0,BYE!" in the log and the surefire plugin fails with:

          The forked VM terminated without saying properly goodbye. VM crash or System.exit called ?
                  at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.close(ForkClient.java:244)
          

          If I check maven/surefire/trunk/surefire-booter/src/main/java/org/apache/maven/surefire/booter/ForkedBooter.java, I see the following lines:

           	runSuitesInProcess( testSet, testClassLoader, startupConfiguration, providerConfiguration );
           	// Say bye.
           	System.out.println("Z,0,BYE!");
           	System.out.flush();
           	// noinspection CallToSystemExit
           	System.exit( 0 );
          

          Why is surefire surprised someone called System.exit() if he's the one who did it?

          Show
          fleiber François LEIBER added a comment - - edited We also have a big issue with surefire 2.12 (using default configuration, which means forkMode=once): the Junit tests of the first module pass perfectly, then we have a "Z,0,BYE!" in the log and the surefire plugin fails with: The forked VM terminated without saying properly goodbye. VM crash or System .exit called ? at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.close(ForkClient.java:244) If I check maven/surefire/trunk/surefire-booter/src/main/java/org/apache/maven/surefire/booter/ForkedBooter.java, I see the following lines: runSuitesInProcess( testSet, testClassLoader, startupConfiguration, providerConfiguration ); // Say bye. System .out.println( "Z,0,BYE!" ); System .out.flush(); // noinspection CallToSystemExit System .exit( 0 ); Why is surefire surprised someone called System.exit() if he's the one who did it?
          Hide
          deivid Deivydas added a comment - - edited

          We have the same issue (-Dtest=test1,test2 doesn't run any test) as Falko (maven 3.0.3/surefire 2.12/junit-4.10) doesn't matter parallel='classes' or parallel='none'. Switching to 2.11 works.

          Show
          deivid Deivydas added a comment - - edited We have the same issue (-Dtest=test1,test2 doesn't run any test) as Falko (maven 3.0.3/surefire 2.12/junit-4.10) doesn't matter parallel='classes' or parallel='none'. Switching to 2.11 works.
          Hide
          krosenvold Kristian Rosenvold added a comment -

          Francois; This is a separate issue from what the others are reporting.

          The lines of code you have found are not the problem, but I am very interested in the actual issue. The code that seems to be failing is new for 2.12, so there would seem to be some kind of unforseen race condition or similar. If you file a separate issue on this one I'd be very interested in your surefire configuration and maybe the log output from running with mvn -X . Does this failure happen all the time ?

          Please file a different ticket.

          Show
          krosenvold Kristian Rosenvold added a comment - Francois; This is a separate issue from what the others are reporting. The lines of code you have found are not the problem, but I am very interested in the actual issue. The code that seems to be failing is new for 2.12, so there would seem to be some kind of unforseen race condition or similar. If you file a separate issue on this one I'd be very interested in your surefire configuration and maybe the log output from running with mvn -X . Does this failure happen all the time ? Please file a different ticket.
          Hide
          fleiber François LEIBER added a comment -

          You're right, I read the ticket title a bit quickly, I'm creating another one (and yes, it happens all the time).

          Show
          fleiber François LEIBER added a comment - You're right, I read the ticket title a bit quickly, I'm creating another one (and yes, it happens all the time).
          Hide
          famod Falko Modler added a comment -

          This very basic maven project can be used to reproduce the issue.

          It has one simple Test (surefire.bug_827.SomeTest) which runs just fine when executing mvn test.

          But when mvn test -Dtest=SomeTest is executed, surefire raises an error: No tests were executed!

          Junit 4.10 is defined in the pom.xml.
          I used Maven 3.0.4 and JDK6u29 on Win7 Home Premium x64.

          When going back to surefire 2.11, everything is fine!

          Show
          famod Falko Modler added a comment - This very basic maven project can be used to reproduce the issue. It has one simple Test (surefire.bug_827.SomeTest) which runs just fine when executing mvn test . But when mvn test -Dtest=SomeTest is executed, surefire raises an error: No tests were executed! Junit 4.10 is defined in the pom.xml. I used Maven 3.0.4 and JDK6u29 on Win7 Home Premium x64. When going back to surefire 2.11, everything is fine!
          Hide
          krosenvold Kristian Rosenvold added a comment -

          The problem is reproduced and "in progress", although not entirely trivial to solve.

          AFAIK this only applies when using the 4.7 JUnit provider, so unless someone has additional info, there is no need to report any more testcases.

          Show
          krosenvold Kristian Rosenvold added a comment - The problem is reproduced and "in progress", although not entirely trivial to solve. AFAIK this only applies when using the 4.7 JUnit provider, so unless someone has additional info, there is no need to report any more testcases.
          Hide
          akuhtz Andreas Kuhtz added a comment -

          Works with current 2.13-SNAPSHOT (maven-surefire-plugin-2.13-20120228.152623-12 ....) with testNG.

          Show
          akuhtz Andreas Kuhtz added a comment - Works with current 2.13-SNAPSHOT (maven-surefire-plugin-2.13-20120228.152623-12 ....) with testNG.
          Hide
          gaul Andrew Gaul added a comment -

          2.13-SNAPSHOT (2.13-20120228.152625-12) still broken with JUnit.

          Show
          gaul Andrew Gaul added a comment - 2.13-SNAPSHOT (2.13-20120228.152625-12) still broken with JUnit.
          Hide
          sebb@apache.org Sebb added a comment -

          I can confirm that the problem is not related to the JUnit version.
          SureFire 2.11 works fine with JUnit 4.7, 4.8, 4.9, 4.10 (and even 3.8.2)
          SureFire 2.12 does not work with any of them.
          For me, the -Dparallel setting makes no difference to 2.12.

          Show
          sebb@apache.org Sebb added a comment - I can confirm that the problem is not related to the JUnit version. SureFire 2.11 works fine with JUnit 4.7, 4.8, 4.9, 4.10 (and even 3.8.2) SureFire 2.12 does not work with any of them. For me, the -Dparallel setting makes no difference to 2.12.
          Hide
          famod Falko Modler added a comment -

          Any progress? In my opinion this is a showstopper (for 2.13).

          Show
          famod Falko Modler added a comment - Any progress? In my opinion this is a showstopper (for 2.13).
          Hide
          gcatania Gabriele Catania added a comment -

          It works on my linux installation, but doesn't on my colleagues' windows ones.

          we are using TestNG with surefire 2.12.

          Show
          gcatania Gabriele Catania added a comment - It works on my linux installation, but doesn't on my colleagues' windows ones. we are using TestNG with surefire 2.12.
          Hide
          krosenvold Kristian Rosenvold added a comment -

          This issue was fixed by the fix for SUREFIRE-824, which re-added support for redmond-based OS'es.

          Show
          krosenvold Kristian Rosenvold added a comment - This issue was fixed by the fix for SUREFIRE-824 , which re-added support for redmond-based OS'es.
          Hide
          krosenvold Kristian Rosenvold added a comment - - edited

          ANd yes, it's fixed on trunk ATM even though 824 is still open.

          Show
          krosenvold Kristian Rosenvold added a comment - - edited ANd yes, it's fixed on trunk ATM even though 824 is still open.
          Hide
          gaul Andrew Gaul added a comment -

          This still fails for me with 2.13-20120620.212637-15 running on Ubuntu 12.04.

          Show
          gaul Andrew Gaul added a comment - This still fails for me with 2.13-20120620.212637-15 running on Ubuntu 12.04.
          Hide
          famod Falko Modler added a comment -

          Works on Windows 7 x64.

          Show
          famod Falko Modler added a comment - Works on Windows 7 x64.
          Hide
          krosenvold Kristian Rosenvold added a comment -

          @Andrew; I am unable to reproduce this on my linux mint installation. Unless you can supply a mini test-project (maybe modify the one already attached) that demonstrates this I will close this issue as cannot reproduce.

          All the ms-windows users chiming in on this issue should note this has been fixed for windows.

          FWIW I tried this pom:

          <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
          <modelVersion>4.0.0</modelVersion>
          <groupId>SUREFIRE</groupId>
          <artifactId>BUG-827</artifactId>
          <version>0.0.1-SNAPSHOT</version>
          <dependencies>
          <dependency>
          <groupId>junit</groupId>
          <artifactId>junit</artifactId>
          <version>4.10</version>
          </dependency>
          </dependencies>
          <properties><surefire.version>2.12</surefire.version></properties>
          <build>
          <pluginManagement>
          <plugins>
          <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-surefire-plugin</artifactId>
          <version>$

          {surefire.version}

          </version>
          <configuration>
          <parallel>classes</parallel>
          <threadCount>2</threadCount>
          </configuration>
          </plugin>
          </plugins>
          </pluginManagement>
          </build>
          </project>

          Show
          krosenvold Kristian Rosenvold added a comment - @Andrew; I am unable to reproduce this on my linux mint installation. Unless you can supply a mini test-project (maybe modify the one already attached) that demonstrates this I will close this issue as cannot reproduce. All the ms-windows users chiming in on this issue should note this has been fixed for windows. FWIW I tried this pom: <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd "> <modelVersion>4.0.0</modelVersion> <groupId>SUREFIRE</groupId> <artifactId>BUG-827</artifactId> <version>0.0.1-SNAPSHOT</version> <dependencies> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.10</version> </dependency> </dependencies> <properties><surefire.version>2.12</surefire.version></properties> <build> <pluginManagement> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <version>$ {surefire.version} </version> <configuration> <parallel>classes</parallel> <threadCount>2</threadCount> </configuration> </plugin> </plugins> </pluginManagement> </build> </project>
          Hide
          gaul Andrew Gaul added a comment -

          Kristian, going back to my example on 06 Feb 2012 with Apache commons-lang 3.1:

          $ mvn test -Dtest=ImmutablePairTest#testBasic -Dparallel=classes -Dsurefire.version=2.11
          ...
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

          $ mvn test -Dtest=ImmutablePairTest#testBasic -Dparallel=classes -Dsurefire.version=2.12
          ...
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0

          $ mvn test -Dtest=ImmutablePairTest#testBasic -Dparallel=classes -Dsurefire.version=2.13-SNAPSHOT
          ...
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0

          Tested with maven-surefire-plugin-2.13-20120706.132117-19.jar .

          Show
          gaul Andrew Gaul added a comment - Kristian, going back to my example on 06 Feb 2012 with Apache commons-lang 3.1: $ mvn test -Dtest=ImmutablePairTest#testBasic -Dparallel=classes -Dsurefire.version=2.11 ... Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 $ mvn test -Dtest=ImmutablePairTest#testBasic -Dparallel=classes -Dsurefire.version=2.12 ... Tests run: 6, Failures: 0, Errors: 0, Skipped: 0 $ mvn test -Dtest=ImmutablePairTest#testBasic -Dparallel=classes -Dsurefire.version=2.13-SNAPSHOT ... Tests run: 6, Failures: 0, Errors: 0, Skipped: 0 Tested with maven-surefire-plugin-2.13-20120706.132117-19.jar .
          Hide
          krosenvold Kristian Rosenvold added a comment -

          This issue was caused by the ClassDemarcatingRunne introduced in SUREFIRE-800, which cased a whole host of issues. Removed CDR

          Show
          krosenvold Kristian Rosenvold added a comment - This issue was caused by the ClassDemarcatingRunne introduced in SUREFIRE-800 , which cased a whole host of issues. Removed CDR
          Hide
          krosenvold Kristian Rosenvold added a comment -

          Fixed in r1364159

          Show
          krosenvold Kristian Rosenvold added a comment - Fixed in r1364159
          Hide
          gaul Andrew Gaul added a comment -

          Verified fixed in r1364394. Thank you for your persistence Kristian.

          Show
          gaul Andrew Gaul added a comment - Verified fixed in r1364394. Thank you for your persistence Kristian.
          Hide
          quixsilver Quintin Siebers added a comment -

          I'm seeing the this behavior with surefire 2.19 and junit 4.12 (also 4.4):

          [INFO] — maven-surefire-plugin:2.19:test (default-cli) @ *removed* —
          No tests were executed!

          [INFO] — maven-surefire-plugin:2.18.1:test (default-cli) @ *removed* —
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

          This is not a complex project, although it is part of a multiproject build.

          MacOSX 10.11.2
          Maven 3.2.5
          Java 1.7.0_71, oracle

          Show
          quixsilver Quintin Siebers added a comment - I'm seeing the this behavior with surefire 2.19 and junit 4.12 (also 4.4): [INFO] — maven-surefire-plugin:2.19:test (default-cli) @ * removed * — No tests were executed! [INFO] — maven-surefire-plugin:2.18.1:test (default-cli) @ * removed * — Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 This is not a complex project, although it is part of a multiproject build. MacOSX 10.11.2 Maven 3.2.5 Java 1.7.0_71, oracle
          Hide
          tibor17 Tibor Digana added a comment -

          Quintin Siebers
          Use version surefire 2.19.1. I can hardly tell you about your issue but you can always post a zip file to jira. Sometimes it is user error. Version 2.19.1 is stable and fixed such test filtering issues as well.

          Show
          tibor17 Tibor Digana added a comment - Quintin Siebers Use version surefire 2.19.1. I can hardly tell you about your issue but you can always post a zip file to jira. Sometimes it is user error. Version 2.19.1 is stable and fixed such test filtering issues as well.
          Hide
          jaroslav.zaruba@gmail.com Jaroslav Záruba added a comment -

          Experiencing this with surefire 2.19.1 and junit 4.11
          After switching to 2.11 tests are executed as they should

          Show
          jaroslav.zaruba@gmail.com Jaroslav Záruba added a comment - Experiencing this with surefire 2.19.1 and junit 4.11 After switching to 2.11 tests are executed as they should
          Hide
          houcem Houcem added a comment -

          I'm experiencing this on ubuntu 16.04 with
          Maven 3.3.9
          Java 1.8.0_131
          Surefire 1.19.1 & 1.20
          the tests don't pass, although they pass on a windows machine

          Show
          houcem Houcem added a comment - I'm experiencing this on ubuntu 16.04 with Maven 3.3.9 Java 1.8.0_131 Surefire 1.19.1 & 1.20 the tests don't pass, although they pass on a windows machine
          Hide
          tibor17 Tibor Digana added a comment -

          Houcem
          Pls attach a small project to reproduce this issue including CLI commands you run.

          Show
          tibor17 Tibor Digana added a comment - Houcem Pls attach a small project to reproduce this issue including CLI commands you run.
          Hide
          tibor17 Tibor Digana added a comment -

          Andrew Gaul
          Houcem
          Jaroslav Záruba
          Quintin Siebers
          In previous Jira comments you said that filtering of test via -Dtest=... does not work properly in Surefire.
          I can help you and resolve this if you send me relevant data. This means a command you run in maven and bunch of tests or sample code attached in this Jira issue
          https://issues.apache.org/jira/browse/SUREFIRE-827

          Current attachment [1] is incorrect because it does not contain expected class(es).

          [1] https://issues.apache.org/jira/secure/attachment/12720135/BUG-827.zip

          Show
          tibor17 Tibor Digana added a comment - Andrew Gaul Houcem Jaroslav Záruba Quintin Siebers In previous Jira comments you said that filtering of test via -Dtest=... does not work properly in Surefire. I can help you and resolve this if you send me relevant data. This means a command you run in maven and bunch of tests or sample code attached in this Jira issue https://issues.apache.org/jira/browse/SUREFIRE-827 Current attachment [1] is incorrect because it does not contain expected class(es). [1] https://issues.apache.org/jira/secure/attachment/12720135/BUG-827.zip

            People

            • Assignee:
              krosenvold Kristian Rosenvold
              Reporter:
              gaul Andrew Gaul
            • Votes:
              26 Vote for this issue
              Watchers:
              27 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development