Uploaded image for project: 'Maven Surefire'
  1. Maven Surefire
  2. SUREFIRE-1925

Bogus skipped test reported for class tested immediately after a JUnit 5 @Disabled class

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0-M4, 3.0.0-M5
    • Fix Version/s: None
    • Labels:
      None
    • Environment:
      64-bit HP Workstation running Ubuntu 20.04.2.
      Java version: openjdk version "11.0.11".
      Maven version: 3.8.1.
      Junit versions: 5.6.0 and 5.8.0-M1 both produce the issue.
    • Docs Text:
      Hide
      Relevant components of the POM.xml file are:

          <properties>
              <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
              <maven.compiler.source>11</maven.compiler.source>
              <maven.compiler.target>11</maven.compiler.target>
              <junit.version>5.8.0-M1</junit.version>
          </properties>

          <build>
              <plugins>
                  <plugin>
                      <artifactId>maven-surefire-plugin</artifactId>
                      <version>3.0.0-M5</version>
                      <configuration>
                          <excludes>
                              <exclude></exclude>
                          </excludes>
                      </configuration>
                  </plugin>
              </plugins>
          </build>

          <dependencies>
              <dependency>
                  <groupId>org.junit.jupiter</groupId>
                  <artifactId>junit-jupiter-api</artifactId>
                  <version>${junit.version}</version>
                  <scope>test</scope>
              </dependency>
              <dependency>
                  <groupId>org.junit.jupiter</groupId>
                  <artifactId>junit-jupiter-engine</artifactId>
                  <version>${junit.version}</version>
                  <scope>test</scope>
              </dependency>
              <dependency>
                  <groupId>org.junit.jupiter</groupId>
                  <artifactId>junit-jupiter-params</artifactId>
                  <version>${junit.version}</version>
                  <scope>test</scope>
              </dependency>
          </dependencies>
      Show
      Relevant components of the POM.xml file are:     <properties>         <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>         <maven.compiler.source>11</maven.compiler.source>         <maven.compiler.target>11</maven.compiler.target>         <junit.version>5.8.0-M1</junit.version>     </properties>     <build>         <plugins>             <plugin>                 <artifactId>maven-surefire-plugin</artifactId>                 <version>3.0.0-M5</version>                 <configuration>                     <excludes>                         <exclude></exclude>                     </excludes>                 </configuration>             </plugin>         </plugins>     </build>     <dependencies>         <dependency>             <groupId>org.junit.jupiter</groupId>             <artifactId>junit-jupiter-api</artifactId>             <version>${junit.version}</version>             <scope>test</scope>         </dependency>         <dependency>             <groupId>org.junit.jupiter</groupId>             <artifactId>junit-jupiter-engine</artifactId>             <version>${junit.version}</version>             <scope>test</scope>         </dependency>         <dependency>             <groupId>org.junit.jupiter</groupId>             <artifactId>junit-jupiter-params</artifactId>             <version>${junit.version}</version>             <scope>test</scope>         </dependency>     </dependencies>

      Description

      If a test class is annotated with org.junit.jupiter.api.Disabled (and is therefore not tested) then the next test class that Maven processes will be reported as having a skipped test that does not in fact exist. This manifests in the .txt file for the second test class in the surefire-reports folder. For example, if the second test class has precisely two methods annotated with org.junit.jupiter.api.Test, neither of which is annotated with org.junit.jupiter.api.Disabled, and if both tests pass then the test report will be like so:

      -------------------------------------------------------------------------------
      Test set: com.example.SecondTest
      -------------------------------------------------------------------------------
      Tests run: 3, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.025 s - in com.example.SecondTest

      It does not matter whether or not the second test class is in the same package as the first. If the two test methods in the second test class are annotated with Disabled, then the Skipped count is reported as 3.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              Labalius Benjamin Morgan
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: