[SUREFIRE-1841] <suiteXmlFiles> no longer finds tests starting with 3.0.0-M4

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0-M4
• Component/s: TestNG support
• Assignee: Unassigned
• Reporter: BJ Cardon
• Created: 13/Sep/20 19:11
• Updated: 20/Dec/21 21:21

Description

Hello. I have a fairly simple multi-module project where I'm using the surefire plugin to execute a series of integration tests in one of the modules. I am doing this by defining a profile in the module that runs one of my suite XML files.

Here is my profile definition in the POM.

<profile>
    <id>acceptance</id>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>3.0.0-M4</version>
                <configuration>
                    <useFile>false</useFile>
                    <suiteXmlFiles>
                        <suiteXmlFile>src/test/resources/jenkins/acceptance-testng.xml</suiteXmlFile>
                    </suiteXmlFiles>
                </configuration>
            </plugin>
        </plugins>
    </build>
</profile>

In previous versions of the plugin, including in 3.0.0-M3, this works and the suite is found and executed. In 3.0.0-M4, the test suite is not found and the tests don't run at all. The same behavior is observed in 3.0.0-M5.

Here is my `acceptance-testng.xml` file

<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd" >
<suite name="Acceptance Test Suite" verbose="1">
    <test name="Group 1 Acceptance Tests" parallel="methods" thread-count="4">
        <groups>
            <run>
                <!-- to exclude a test from Acceptance test run, add test to one of these groups -->
                <exclude name="fragile"/>
                <exclude name="load"/>
                
                <!-- This testuite will only run tests that belong to the 'acceptance' group -->
                <include name="acceptance"/>
            </run>
        </groups>
        <classes>
            <class name="com.company.app.integration.suite.rest.AppRestTest"/>
            <class name="com.company.app.integration.suite.rest.AppRestErrorTest"/>
            <class name="com.company.app.integration.suite.rest.AppAuthTest"/>
            <class name="com.company.app.integration.suite.rest.AppLoadTest"/>
        </classes>
    </test>

    <test name="Group 2 Acceptance Tests" parallel="classes" thread-count="4">
        <groups>
            <run>
                <!-- to exclude a test from Acceptance test run, add test to one of these groups -->
                <exclude name="fragile"/>
                <exclude name="load"/>                

                <!-- This testuite will only run tests that belong to the 'acceptance' group -->
                <include name="acceptance"/>
            </run>
        </groups>
        <classes>
            <class name="com.company.app.integration.suite.vendor.VendorRestTest"/>
            <class name="com.company.app.integration.suite.vendor.VendorRestErrorTest"/>
            <class name="com.company.app.integration.suite.vendor.VendorAuthTest"/>
            <class name="com.company.app.integration.suite.vendor.VendorLoadTest"/>
        </classes>
    </test>
</suite>

As stated above, this broke sometime between 3.0.0-M3 and 3.0.0-M4. I tried looking through the changelog (on a tangent, the changelogs linked from the ASF page require creating an account in Jira which seems odd) but it seems like a number of fairly large changes were made in 3.0.0-M4 that could have been to blame for this change in behavior.

Let me know if I can provide any additional context.

Issue Links

• is related to: SUREFIRE-1885 Maven is not running any tests called using testng suitexml files in POM