[SUREFIRE-1925] Bogus skipped test reported for class tested immediately after a JUnit 5 @Disabled class

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0-M4, 3.0.0-M5
• Component/s: JUnit 5.x support, Maven Surefire Plugin
• Environment: 64-bit HP Workstation running Ubuntu 20.04.2. ; Java version: openjdk version "11.0.11". ; Maven version: 3.8.1. ; Junit versions: 5.6.0 and 5.8.0-M1 both produce the issue.

Docs Text

Relevant components of the POM.xml file are:

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <maven.compiler.source>11</maven.compiler.source>
        <maven.compiler.target>11</maven.compiler.target>
        <junit.version>5.8.0-M1</junit.version>
    </properties>

    <build>
        <plugins>
            <plugin>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>3.0.0-M5</version>
                <configuration>
                    <excludes>
                        <exclude></exclude>
                    </excludes>
                </configuration>
            </plugin>
        </plugins>
    </build>

    <dependencies>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-api</artifactId>
            <version>${junit.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-engine</artifactId>
            <version>${junit.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-params</artifactId>
            <version>${junit.version}</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

• Assignee: Unassigned
• Reporter: Benjamin Morgan
• Created: 02/Jul/21 21:56
• Updated: 02/Jul/21 21:56

Description

If a test class is annotated with org.junit.jupiter.api.Disabled (and is therefore not tested) then the next test class that Maven processes will be reported as having a skipped test that does not in fact exist. This manifests in the .txt file for the second test class in the surefire-reports folder. For example, if the second test class has precisely two methods annotated with org.junit.jupiter.api.Test, neither of which is annotated with org.junit.jupiter.api.Disabled, and if both tests pass then the test report will be like so:

-------------------------------------------------------------------------------
Test set: com.example.SecondTest
-------------------------------------------------------------------------------
Tests run: 3, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.025 s - in com.example.SecondTest

It does not matter whether or not the second test class is in the same package as the first. If the two test methods in the second test class are annotated with Disabled, then the Skipped count is reported as 3.