Maven Surefire
  1. Maven Surefire
  2. SUREFIRE-377

When JUnit and TestNG tests are in same project, only one set gets run

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.4
    • Fix Version/s: 2.4
    • Component/s: TestNG support
    • Labels:
      None

      Description

      The attached Maven project has two tests: one JUnit test and one TestNG test. According to the documentation, in this case TestNG should run both tests.

      Run "mvn test". Only the TestNG test will run. If you modify the pom to set the property "junit=true", only the JUnit test will run.

      <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-surefire-plugin</artifactId>
      <version>2.4-SNAPSHOT</version>
      <configuration>
      <properties>
      <property>
      <name>junit</name>
      <value>true</value>
      </property>
      </properties>
      </plugin>

      1. surefire377.patch
        8 kB
        Dan Fabulich
      2. testng-junit-together.zip
        3 kB
        Dan Fabulich

        Issue Links

          Activity

          Hide
          Dan Fabulich added a comment -

          IMO, this problem is probably too hard to be worth solving. See the mailing list archive thread here:

          http://mail-archives.apache.org/mod_mbox/maven-surefire-dev/200711.mbox/%3cPine.WNT.4.64.0711191635230.2484@sfeng02.rf.lan%3e
          http://tinyurl.com/32745g

          Alex suggested a good way to tell the different between JUnit and TestNG runs, but trying to dynamically construct multiple test runs from two arrays of classes that don't clobber each other when you go to run them is harder than it has any right to be.

          Show
          Dan Fabulich added a comment - IMO, this problem is probably too hard to be worth solving. See the mailing list archive thread here: http://mail-archives.apache.org/mod_mbox/maven-surefire-dev/200711.mbox/%3cPine.WNT.4.64.0711191635230.2484@sfeng02.rf.lan%3e http://tinyurl.com/32745g Alex suggested a good way to tell the different between JUnit and TestNG runs, but trying to dynamically construct multiple test runs from two arrays of classes that don't clobber each other when you go to run them is harder than it has any right to be.
          Hide
          Dan Fabulich added a comment -

          I'm going to revert my workspace to blow away my current work on this; in the meantime, here's the current check-in. It sort-of works, but the TestNG XML output is clobbered when you run TestNG twice; I don't see a good way to fix that.

          Show
          Dan Fabulich added a comment - I'm going to revert my workspace to blow away my current work on this; in the meantime, here's the current check-in. It sort-of works, but the TestNG XML output is clobbered when you run TestNG twice; I don't see a good way to fix that.
          Hide
          Dan Fabulich added a comment -

          Fixed in revision 598205. Basically just the patch as was, but we drop testng results in separate directories if we need to run both junit and testng results at once.

          Show
          Dan Fabulich added a comment - Fixed in revision 598205. Basically just the patch as was, but we drop testng results in separate directories if we need to run both junit and testng results at once.
          Hide
          Pawel Poltorak added a comment - - edited

          The fix does not work if tests are written using junit 4.4

          Perhaps with additional check it would work:
          ...
          if (junit.framework.Test.class.isAssignableFrom( c ) || isJunit4(c))

          { junitTestClasses.add( c ); }

          else

          { testNgTestClasses.add( c ); }

          ...

          public boolean isJunit4(Class c) {
          Class<? extends Annotation> testAnnotation = null;
          try

          { testAnnotation = (Class<? extends Annotation>) Class.forName("org.junit.Test"); }

          catch (ClassNotFoundException e)

          { return false; }

          Method[] methods = c.getMethods();
          for (int i = 0; i < methods.length; i++) {
          if (methods[i].isAnnotationPresent(testAnnotation))

          { return true; }

          }
          return false;
          }

          Show
          Pawel Poltorak added a comment - - edited The fix does not work if tests are written using junit 4.4 Perhaps with additional check it would work: ... if (junit.framework.Test.class.isAssignableFrom( c ) || isJunit4(c)) { junitTestClasses.add( c ); } else { testNgTestClasses.add( c ); } ... public boolean isJunit4(Class c) { Class<? extends Annotation> testAnnotation = null; try { testAnnotation = (Class<? extends Annotation>) Class.forName("org.junit.Test"); } catch (ClassNotFoundException e) { return false; } Method[] methods = c.getMethods(); for (int i = 0; i < methods.length; i++) { if (methods [i] .isAnnotationPresent(testAnnotation)) { return true; } } return false; }
          Hide
          Rune Engseth added a comment -

          I agree with Pawel

          When adopting TestNG into an existing project with lots of existing JUnit 4.4 tests, it would be nice if both types of tests could be run at once.

          If Pawel's additional check would fix this, is it possible to add this to the next SNAPSHOT version?

          Show
          Rune Engseth added a comment - I agree with Pawel When adopting TestNG into an existing project with lots of existing JUnit 4.4 tests, it would be nice if both types of tests could be run at once. If Pawel's additional check would fix this, is it possible to add this to the next SNAPSHOT version?
          Hide
          Roger Deloy Pack added a comment -

          I think I just got bit with this one, too.
          Is this still broken with latest versions then? [i.e. junit 4.4 aren't run at all when packaged with testng classes]?
          In my instance it was one module [junit] depending on another [testng] and thus the firsts tests aren't being run. Sniff.
          -=r

          Show
          Roger Deloy Pack added a comment - I think I just got bit with this one, too. Is this still broken with latest versions then? [i.e. junit 4.4 aren't run at all when packaged with testng classes] ? In my instance it was one module [junit] depending on another [testng] and thus the firsts tests aren't being run. Sniff. -=r
          Hide
          Roger Deloy Pack added a comment -

          In my case it might have been being caused by:
          <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-surefire-plugin</artifactId>
          <version>2.1</version>
          </plugin>

          in my pom.xml [note the old version]

          Show
          Roger Deloy Pack added a comment - In my case it might have been being caused by: <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <version>2.1</version> </plugin> in my pom.xml [note the old version]
          Hide
          Pawel Poltorak added a comment -

          Hi,

          I'm afraid there is more to be done to allow both JUnit and TestNG tests. Changing surefire plugin is the easy part. Surefire uses testng runner to run JUnit tests, which doesn't support JUnit4 tests. Therefore TestNG team would have to change their runner to support JUnit4

          Actually, I've made changes to both libraries. If there is someone from surefire team willing to apply my changes I can generate a patch. I've also contacted TestNG team, but they are not 100% sure they want to do junit4 support.

          Show
          Pawel Poltorak added a comment - Hi, I'm afraid there is more to be done to allow both JUnit and TestNG tests. Changing surefire plugin is the easy part. Surefire uses testng runner to run JUnit tests, which doesn't support JUnit4 tests. Therefore TestNG team would have to change their runner to support JUnit4 Actually, I've made changes to both libraries. If there is someone from surefire team willing to apply my changes I can generate a patch. I've also contacted TestNG team, but they are not 100% sure they want to do junit4 support.
          Hide
          James Kato added a comment -

          Hi,
          I am trying to run both JUnit 4.4 and TestNG 5.8 tests in the same suite. I seem to have come accross this problem in surefire 2.4.3 where as soon as I add testNG as a dependency, the TestNG runner is used. The problem is that this runner cannot handle JUnit 4.4 tests: I get:

          Running TestSuite
          org.testng.TestNGException:
          Failure in JUnit mode for class com.sampleTest: could not create/run JUnit test suite:
          cannot retrieve JUnit method

          Which is resolved if I extend TestCase in com.sampleTest as in JUnit 3.x

          A sample of the pom.xml I use is:

          <dependencies>
          <dependency>
          <groupId>org.testng</groupId>
          <artifactId>testng</artifactId>
          <version>5.8</version>
          <scope>test</scope>
          <classifier>jdk15</classifier>
          </dependency>
          <dependency>
          <groupId>junit</groupId>
          <artifactId>junit</artifactId>
          <version>4.4</version>
          <scope>test</scope>
          </dependency>
          </dependencies>
          <build>
          <plugins>
          <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-surefire-plugin</artifactId>
          <version>2.4.3</version>
          <configuration>
          <properties>
          <property>
          <name>junit</name>
          <value>true</value>
          </property>
          </properties>
          </configuration>
          </plugin>
          </plugins>
          </build>

          Note, at present I would be happy to run all JUnit 4.4 tests in one execution and then use a profile to run all TestNG tests.

          Please can you tell me whether you know of a work around for this?

          Cheers,
          James

          Show
          James Kato added a comment - Hi, I am trying to run both JUnit 4.4 and TestNG 5.8 tests in the same suite. I seem to have come accross this problem in surefire 2.4.3 where as soon as I add testNG as a dependency, the TestNG runner is used. The problem is that this runner cannot handle JUnit 4.4 tests: I get: Running TestSuite org.testng.TestNGException: Failure in JUnit mode for class com.sampleTest: could not create/run JUnit test suite: cannot retrieve JUnit method Which is resolved if I extend TestCase in com.sampleTest as in JUnit 3.x A sample of the pom.xml I use is: <dependencies> <dependency> <groupId>org.testng</groupId> <artifactId>testng</artifactId> <version>5.8</version> <scope>test</scope> <classifier>jdk15</classifier> </dependency> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.4</version> <scope>test</scope> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <version>2.4.3</version> <configuration> <properties> <property> <name>junit</name> <value>true</value> </property> </properties> </configuration> </plugin> </plugins> </build> Note, at present I would be happy to run all JUnit 4.4 tests in one execution and then use a profile to run all TestNG tests. Please can you tell me whether you know of a work around for this? Cheers, James
          Hide
          James Kato added a comment -

          Well, I've found a work around which is not too painful, just a little awkward.

          Because it is the TestNG dependency that triggers surefire to use the TestNG runner to execute JUnit tests, I've moved this dependency out of the main project scope. In order to compile and run all JUnit tests, I exclude the TestNG tests from the compiler and surefire plugins.

          Then in a profile, I add my TestNG dependency and adjust the compiler and surefire plugins to include the TestNG tests and the test suite file.

          To run JUnit tests I use: mvn test
          To run testNG tests I use: mvn test -P testng-profile

          Simple when you know how. Any better solutions appreciated since I see problems with this as my number of tests increases.

          James

          Show
          James Kato added a comment - Well, I've found a work around which is not too painful, just a little awkward. Because it is the TestNG dependency that triggers surefire to use the TestNG runner to execute JUnit tests, I've moved this dependency out of the main project scope. In order to compile and run all JUnit tests, I exclude the TestNG tests from the compiler and surefire plugins. Then in a profile, I add my TestNG dependency and adjust the compiler and surefire plugins to include the TestNG tests and the test suite file. To run JUnit tests I use: mvn test To run testNG tests I use: mvn test -P testng-profile Simple when you know how. Any better solutions appreciated since I see problems with this as my number of tests increases. James
          Hide
          Udai Gupta added a comment -

          Shouldn't this bug's resolution should be "won't resolve" or something. If it's fixed can someone please help me to understand how it is fixed(because fixing this bug means both Junitand TestNG tests should run in one set, by some auto discovery mechanism internally)

          Show
          Udai Gupta added a comment - Shouldn't this bug's resolution should be "won't resolve" or something. If it's fixed can someone please help me to understand how it is fixed(because fixing this bug means both Junitand TestNG tests should run in one set, by some auto discovery mechanism internally)
          Hide
          Pawel Poltorak added a comment -

          Udai, check my comment above. Surefire uses TestNG library to run junit tests and TestNG doesn't support running junit4 (only junit3). That's why junit4 tests are not being run.

          Show
          Pawel Poltorak added a comment - Udai, check my comment above. Surefire uses TestNG library to run junit tests and TestNG doesn't support running junit4 (only junit3). That's why junit4 tests are not being run.
          Hide
          Udai Gupta added a comment -

          Thanks Pawel, I guess I might need a Junit 4 to Junit3 migration tool

          Show
          Udai Gupta added a comment - Thanks Pawel, I guess I might need a Junit 4 to Junit3 migration tool
          Hide
          Aleksei Valikov added a comment - - edited

          I have a better solution.

          The idea is to create two executions of the maven-surefire-plugin, one for JUnit, one for TestNG. You can disable one of JUnit/TestNG per execution by specifying nonexisting junitArtifactName or testNGArtifactName:

          <plugin>
          	<groupId>org.apache.maven.plugins</groupId>
          	<artifactId>maven-surefire-plugin</artifactId>
          	<executions>
          		<execution>
          			<phase>test</phase>
          			<goals>
          				<goal>test</goal>
          			</goals>
          			<configuration>	
          				<testNGArtifactName>none:none</testNGArtifactName>
          			</configuration>
          		</execution>
          		<execution>
          			<id>test-testng</id>
          			<phase>test</phase>
          			<goals>
          				<goal>test</goal>
          			</goals>
          			<configuration>	
          				<junitArtifactName>none:none</junitArtifactName>
          			</configuration>
          		</execution>
          	</executions>
          </plugin>
          
          Show
          Aleksei Valikov added a comment - - edited I have a better solution . The idea is to create two executions of the maven-surefire-plugin, one for JUnit, one for TestNG. You can disable one of JUnit/TestNG per execution by specifying nonexisting junitArtifactName or testNGArtifactName : <plugin> <groupId> org.apache.maven.plugins </groupId> <artifactId> maven-surefire-plugin </artifactId> <executions> <execution> <phase> test </phase> <goals> <goal> test </goal> </goals> <configuration> <testNGArtifactName> none:none </testNGArtifactName> </configuration> </execution> <execution> <id> test-testng </id> <phase> test </phase> <goals> <goal> test </goal> </goals> <configuration> <junitArtifactName> none:none </junitArtifactName> </configuration> </execution> </executions> </plugin>
          Hide
          James Kato added a comment - - edited

          Aleksei's fix works well, thanks! I just had to mess about with the default execution, so here is a variation using Aleksei's solution.

          <plugin>
          	<groupId>org.apache.maven.plugins</groupId>
          	<artifactId>maven-surefire-plugin</artifactId>
          	<version>2.4.3</version>
          	<configuration>	          
                          <testNGArtifactName>none:none</testNGArtifactName>   
          		<excludes>
          			<exclude>**/integration/*.java</exclude>
          		</excludes> 
                  </configuration>
          	<executions>
          		<execution>
          			<id>integration-test</id>
          			<goals>
          				<goal>test</goal>
          			</goals>
          			<phase>integration-test</phase>
          			<configuration>
          				<junitArtifactName>none:none</junitArtifactName>
          				<testNGArtifactName>org.testng:testng</testNGArtifactName>
          				<suiteXmlFiles>
          					<suiteXmlFile>
                                                     src/test/resourcestestng.xml
                                                  </suiteXmlFile>
          				</suiteXmlFiles>
          				<excludes>
          					<exclude>none</exclude>
          				</excludes>
          				<includes>
          					<include>**/integration/*.java</include>
          				</includes>
          			</configuration>
          		</execution>
          	</executions>
          </plugin>
          
          Show
          James Kato added a comment - - edited Aleksei's fix works well, thanks! I just had to mess about with the default execution, so here is a variation using Aleksei's solution. <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <version>2.4.3</version> <configuration> <testNGArtifactName>none:none</testNGArtifactName> <excludes> <exclude>**/integration/*.java</exclude> </excludes> </configuration> <executions> <execution> <id>integration-test</id> <goals> <goal>test</goal> </goals> <phase>integration-test</phase> <configuration> <junitArtifactName>none:none</junitArtifactName> <testNGArtifactName>org.testng:testng</testNGArtifactName> <suiteXmlFiles> <suiteXmlFile> src/test/resourcestestng.xml </suiteXmlFile> </suiteXmlFiles> <excludes> <exclude>none</exclude> </excludes> <includes> <include>**/integration/*.java</include> </includes> </configuration> </execution> </executions> </plugin>
          Hide
          Gabriele Kahlout added a comment -

          @Aleksei: thank you, a year from your comment I used your solution verbatim (+ version) and it worked. For the reference, this is the pom.

          Show
          Gabriele Kahlout added a comment - @Aleksei: thank you, a year from your comment I used your solution verbatim (+ version) and it worked. For the reference, this is the pom .
          Hide
          Jim Sellers added a comment -

          Thanks all for the solution. Note: I ran into an issue when using it with sonar. See SONAR-2408

          Show
          Jim Sellers added a comment - Thanks all for the solution. Note: I ran into an issue when using it with sonar. See SONAR-2408
          Hide
          Kay McCormick added a comment -

          I don't understand how this issue is fixed, since I haven't seen or read any indication that JUnit and TestNG tests can be run as part of a single project build. If there is a solution to this I would enjoy hearing it.

          Show
          Kay McCormick added a comment - I don't understand how this issue is fixed, since I haven't seen or read any indication that JUnit and TestNG tests can be run as part of a single project build. If there is a solution to this I would enjoy hearing it.

            People

            • Assignee:
              Unassigned
              Reporter:
              Dan Fabulich
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development