[SUREFIRE-1816] XML Report Is generated incorrect only for failed tests in TestFactory or when rerunFailingTestsCount>0

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0-M4
• Component/s: Maven Failsafe Plugin
• Labels: reporting rerun
• Assignee: Unassigned
• Reporter: Iuri Litvinov
• Created: 07/Jul/20 12:18
• Updated: 08/Jul/20 07:03

Description

We have TestFactory with DynamicTests like in example. If rerunFailingTestsCount>0 and one of the tests are failed, the plugin generates XML with only one test or only passed tests.

If rerun is 0 or all tests passed, everything is ok.

 

TestFactory:

@Tags(Tag("w"))
@Execution(SAME_THREAD)
@TestFactory
fun `Factory tests`() = iterator<DynamicTest> {

    yield(dynamicTest("Test1") {
        assert(false)
    })

    yield(dynamicTest("Test2") {
        assert(true)
    })
}

 

Pom contains:

 

<artifactId>maven-failsafe-plugin</artifactId>
    <version>3.0.0-M4</version>
    <configuration>
        <consoleOutputReporter implementation="org.apache.maven.plugin.surefire.extensions.junit5.JUnit5ConsoleOutputReporter">
            <disable>false</disable>
            <encoding>UTF-8</encoding>
            <usePhrasedFileName>false</usePhrasedFileName>
        </consoleOutputReporter>
        <statelessTestsetInfoReporter implementation="org.apache.maven.plugin.surefire.extensions.junit5.JUnit5StatelessTestsetInfoReporter">
            <disable>false</disable>
            <usePhrasedClassNameInRunning>true</usePhrasedClassNameInRunning>
            <usePhrasedClassNameInTestCaseSummary>true</usePhrasedClassNameInTestCaseSummary>
            <usePhrasedFileName>false</usePhrasedFileName>
        </statelessTestsetInfoReporter>
        <statelessTestsetReporter implementation="org.apache.maven.plugin.surefire.extensions.junit5.JUnit5Xml30StatelessReporter">
            <disable>false</disable>
            <usePhrasedFileName>false</usePhrasedFileName>
            <usePhrasedTestCaseClassName>true</usePhrasedTestCaseClassName>
            <usePhrasedTestCaseMethodName>true</usePhrasedTestCaseMethodName>
            <usePhrasedTestSuiteClassName>true</usePhrasedTestSuiteClassName>
            <version>3.0</version>
        </statelessTestsetReporter>
        <trimStackTrace>false</trimStackTrace>
        <parallel>classes</parallel>
        <forkCount>${fork.count}</forkCount>
        <threadCount>${fork.count}</threadCount>
        <reuseForks>false</reuseForks>
        <argLine>-Xmx2048m</argLine>
        <rerunFailingTestsCount>1</rerunFailingTestsCount>
        <includes>
            <include>${includeRule}</include>
        </includes>
        <excludes>
            <exclude>${excludeRule}</exclude>
        </excludes>
        <groups>${tags}</groups>
        <reportsDirectory>${project.build.directory}/failsafe-reports/${timestamp}</reportsDirectory>
        <systemProperties>
            <property>
                <name>junit.jupiter.extensions.autodetection.enabled</name>
                <value>true</value>
            </property>
        </systemProperties>
        <systemPropertyVariables>
            <osVersion>${osVersion}</osVersion>
            <device>${device}</device>
        </systemPropertyVariables>
    </configuration>
    <executions>
        <execution>
            <goals>
                <goal>integration-test</goal>
                <goal>verify</goal>
            </goals>
        </execution>
    </executions>
</plugin>