[SUREFIRE-1757] maven-surefire-plugin versions starting from 2.22.0 do not run JUnit5 Test Suites junit-platform-runner

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.22.0, 2.22.1, 2.22.2, 3.0.0-M1, 3.0.0-M2, 3.0.0-M3, 3.0.0-M4
• Component/s: JUnit 5.x support, Maven Surefire Plugin
• Assignee: Tibor Digana
• Reporter: Denys Galyuk
• Created: 28/Feb/20 14:47
• Updated: 05/Jun/20 20:36
• Resolved: 05/Jun/20 20:36

Description

plugin works with version 2.19.1 (or 2.20, or 2.21.0)

I've attached the project archive to the bug report.

Preconditions:

pom.xml

1. maven-surefire-plugin 2.19.1 (or 2.20, or 2.21.0)
2. maven-compiler-plugin 3.8.1
3. junit-platform-runner 1.6.0
4. junit-jupiter-engine 5.6.0

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
   <modelVersion>4.0.0</modelVersion>

   <groupId>tag</groupId>
   <artifactId>tag</artifactId>
   <version>1.0-SNAPSHOT</version>

   <dependencies>
       <!-- https://mvnrepository.com/artifact/org.junit.jupiter/junit-jupiter-engine -->
       <dependency>
           <groupId>org.junit.jupiter</groupId>
           <artifactId>junit-jupiter-engine</artifactId>
           <version>5.6.0</version>
           <scope>test</scope>
       </dependency>
       <!-- https://mvnrepository.com/artifact/org.junit.platform/junit-platform-runner -->
       <dependency>
           <groupId>org.junit.platform</groupId>
           <artifactId>junit-platform-runner</artifactId>
           <version>1.6.0</version>
           <scope>test</scope>
       </dependency>
   </dependencies>


   <build>
       <plugins>
           <!-- https://mvnrepository.com/artifact/org.apache.maven.plugins/maven-compiler-plugin -->
           <plugin>
               <groupId>org.apache.maven.plugins</groupId>
               <artifactId>maven-compiler-plugin</artifactId>
               <version>3.8.1</version>
               <configuration>
                   <source>1.8</source>
                   <target>1.8</target>
                   <encoding>UTF-8</encoding>
               </configuration>
           </plugin>
           <!-- https://mvnrepository.com/artifact/org.apache.maven.plugins/maven-surefire-plugin -->
           <plugin>
               <groupId>org.apache.maven.plugins</groupId>
               <artifactId>maven-surefire-plugin</artifactId>
<!-- ALL PLUGIN VERSIONS BELOW DO NOT SEE THE JUNIT TEST SUITE AND AS A RESULT MAVEN FAILS THE BUILD WITH THE ERROR:
    No tests were executed! -->
<!--                <version>2.22.2</version>-->
<!--                <version>2.22.1</version>-->
<!--                <version>2.22.0</version>-->
<!--                <version>2.20.1</version>-->
<!--                <version>3.0.0-M4</version>-->
<!--                <version>3.0.0-M3</version>-->
<!--                <version>3.0.0-M2</version>-->
<!--                <version>3.0.0-M1</version>-->

<!--                WORKING VERSIONS OF THE PLUGIN: JUnit5 Suites (based on tags) + Maven -->
               <version>2.21.0</version>
<!--                <version>2.20</version>-->
<!--                <version>2.19.1</version>-->
           </plugin>
       </plugins>
   </build>
</project>

Test class base on JUnit5: @Test,@Tag,@Tags

package tests.testsuites.suite1;

import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Tags;
import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.assertTrue;

public class Suite2TestNameTagsTests {
   @Test
   @Tag("UAT")
   void suite2_1Test() {
       System.out.println("Suite2TestNameTagsTests:suite2_1Test(UAT tag)");
       assertTrue(true);
   }

   @Test
   @Tag("SMOKE")
   void suite2_2Test() {
       System.out.println("Suite2TestNameTagsTests:suite2_2Test(SMOKE tag)");
       assertTrue(true);
   }

   @Test
   @Tags({@Tag("SMOKE"), @Tag("UAT")})
   void suite2_3Test() {
       System.out.println("Suite2TestNameTagsTests:suite2_3Test(SMOKE and UAT tags)");
       assertTrue(true);
   }
}

JUnit5 Test Suite based on: @RunWith, @SelectPackages, @IncludeTags

package tests.suites;

import org.junit.platform.runner.JUnitPlatform;
import org.junit.platform.suite.api.IncludeTags;
import org.junit.platform.suite.api.SelectPackages;
import org.junit.runner.RunWith;

@RunWith(JUnitPlatform.class)
@SelectPackages("tests")
@IncludeTags({"SMOKE","UAT"})
public class SmokeAndUatSuiteTest {
}

Steps to reproduce: (Run the created test suite with maven)

1. Open terminal or GitBASH in the project folder
2. Run the command mvn clean test -Dtest=TestSuite

Actual Result: (something like this)

BUILD SUCCESS
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0

$ mvn clean test -Dtest=SmokeAndUatSuiteTest
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------< tag:tag >-------------------------------
[INFO] Building tag 1.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ tag ---
[INFO] Deleting C:\Users\galyuk\IdeaProjects\tag\target
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ tag ---
[WARNING] Using platform encoding (Cp1252 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 0 resource
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ tag ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ tag ---
[WARNING] Using platform encoding (Cp1252 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory C:\Users\galyuk\IdeaProjects\tag\src\test\resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ tag ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 11 source files to C:\Users\galyuk\IdeaProjects\tag\target\test-classes
[INFO]
[INFO] --- maven-surefire-plugin:2.21.0:test (default-test) @ tag ---
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running tests.suites.SmokeAndUatSuiteTest
Suite2TestNameTagsTests:suite2_1Test(UAT tag)
Suite2TestNameTagsTests:suite2_2Test(SMOKE tag)
Suite2TestNameTagsTests:suite2_3Test(SMOKE and UAT tags)
Suite3ClassNameTagsTests(SMOKE and UAT tags):suite3_1Test
Suite4TestNameTagsTests(SMOKE tag):suite4_1Test
Suite1ClassNameTagsTests(UAT tag):suite1_1Test
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.595 s - in tests.suites.SmokeAndUatSuiteTest
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  5.915 s
[INFO] Finished at: 2020-02-28T16:17:00+02:00
[INFO] ------------------------------------------------------------------------

plugin does not work with versions 2.22.0 and higher

Preconditions:

1. Change the version of the maven-surefire-plugin in the pom.xml to any from the list below:

   2.20.1
   2.22.0
   2.22.1
   2.22.2
   3.0.0-M1
   3.0.0-M2
   3.0.0-M3
   3.0.0-M4

Steps to reproduce: (Run the created test suite with maven)

1. Open terminal or GitBASH in the project folder
2. Run the command mvn clean test -Dtest=TestSuite

Actual Result: (something like this)

BUILD FAILURE: No tests were executed!

$ mvn clean test -Dtest=SmokeAndUatSuiteTest
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------< tag:tag >-------------------------------
[INFO] Building tag 1.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ tag ---
[INFO] Deleting C:\Users\galyuk\IdeaProjects\tag\target
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ tag ---
[WARNING] Using platform encoding (Cp1252 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 0 resource
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ tag ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ tag ---
[WARNING] Using platform encoding (Cp1252 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory C:\Users\galyuk\IdeaProjects\tag\src\test\resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ tag ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 11 source files to C:\Users\galyuk\IdeaProjects\tag\target\test-classes
[INFO]
[INFO] --- maven-surefire-plugin:2.22.1:test (default-test) @ tag ---
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  5.611 s
[INFO] Finished at: 2020-02-28T16:16:42+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.22.1:test (default-test) on project tag: No tests were executed!  (Set -DfailIfNoTests=false to ignore this error.) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

---

Please fix

Issue Links

• duplicates: SUREFIRE-1787 Support multiple runners (JUnit4, TestNG, other) and their API in JUnit5 Provider
• is duplicated by: SUREFIRE-1528 org.apache.maven.surefire.booter.SurefireBooterForkException with maven-surefire-plugin 2.22 with jdk 10.0.1
• is duplicated by: SUREFIRE-1763 surefire does not run junit5 tests