[SUREFIRE-2180] Surefire not invoking tests with TestNG DataProvider

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.2
• Component/s: TestNG support
• External issue URL: https://github.com/testng-team/testng/issues/2924
• Assignee: Unassigned
• Reporter: Javier A. Ortiz
• Created: 16/Jun/23 14:17
• Updated: 10/Jul/23 21:33

Description

I'm having a weird issue I can't figure out. It's driving me insane.

I have a Maven project for which I'm running the same command both locally and on Jenkins. The environment is the same as far as I can tell:

• Maven 3.9.2 (On Jenkins: Docker image maven:3.9.2-eclipse-temurin-11)
• JDK 11 (On Jenkins: Docker image maven:3.9.2-eclipse-temurin-11)
• TestNG 7.8.0

Command:

mvn -Ddependency.surefire.verbose=10 -Pcoverage verify -B -U -Dgroup=PLATFORM

Relevant POM section:

<build>
    <pluginManagement>
      <plugins>
        <plugin>
          <groupId>org.jacoco</groupId>
          <artifactId>jacoco-maven-plugin</artifactId>
          <version>0.8.10</version>
        </plugin>
        ...
      </plugins>
    </pluginManagement>
...
</build>
...
<profiles>
    <profile>
      <id>coverage</id>
      <build>
        <plugins>
          <plugin>
            <groupId>org.jacoco</groupId>
            <artifactId>jacoco-maven-plugin</artifactId>
            <configuration>
              <excludes>
                <exclude>com/XXX/framework/dataproviders/**/*</exclude>
                <exclude>com/XXX/v2/basetest/**/*</exclude>
                <exclude>**/*Exception*</exclude>
              </excludes>
            </configuration>
            <executions>
              <execution>
                <goals>
                  <goal>prepare-agent</goal>
                </goals>
              </execution>
              <!-- attached to Maven test phase -->
              <execution>
                <id>report</id>
                <phase>test</phase>
                <goals>
                  <goal>report</goal>
                </goals>
              </execution>
              <!-- Add this checking -->
              <execution>
                <id>jacoco-check</id>
                <goals>
                  <goal>check</goal>
                </goals>
                <configuration>
                  <rules>
                    <rule>
                      <element>BUNDLE</element>
                      <limits>
                        <limit>
                          <counter>INSTRUCTION</counter>
                          <value>COVEREDRATIO</value>
                          <minimum>60%</minimum>
                        </limit>
                        <limit>
                          <counter>CLASS</counter>
                          <value>MISSEDCOUNT</value>
                          <maximum>0</maximum>
                        </limit>
                      </limits>
                    </rule>
                    <rule>
                      <element>PACKAGE</element>
                      <limits>
                        <limit>
                          <counter>LINE</counter>
                          <value>COVEREDRATIO</value>
                          <minimum>60%</minimum>
                        </limit>
                      </limits>
                    </rule>
                  </rules>
                  <skip>${skipTests}</skip>
                </configuration>
              </execution>
            </executions>
          </plugin>
        </plugins>
      </build>
    </profile>

One of the tests that have the issue (only tests with DataProviders) show this pattern in the logs. Enabled verbose to level 10 in hopes that something would show up.

Locally:

[2023-06-06 16:55:46.795] [INFO] [TestClass] Creating TestClass for [ClassImpl class=com.XXX.framework.utils.TimeUtilsTest]
[2023-06-06 16:55:46.796] [INFO] Method public java.lang.Object[][] com.XXX.framework.utils.TimeUtilsTest.getTimeData() has a @Test annotation but also a return value: ignoring it. Use <suite allow-return-values="true"> to fix this
[2023-06-06 16:55:46.797] [INFO] [TestClass] Adding method TimeUtilsTest.testToFormattedString(java.util.concurrent.TimeUnit,int,java.lang.String,com.XXX.framework.annotations.TestNameParameter)[pri:0, instance:null] on TestClass class com.XXX.framework.utils.TimeUtilsTest

...

[2023-06-06 16:55:47.316] [INFO] ===== Test class
com.XXX.framework.utils.TimeUtilsTest
[2023-06-06 16:55:47.316] [INFO]     @Test TimeUtilsTest.testToFormattedString(java.util.concurrent.TimeUnit,int,java.lang.String,com.XXX.framework.annotations.TestNameParameter)[pri:0, instance:com.XXX.framework.utils.TimeUtilsTest@283eb984]
[2023-06-06 16:55:47.316] [INFO] ======

...

[TestNG] INVOKING: "Surefire test" - com.XXX.framework.utils.TimeUtilsTest.testToFormattedString(java.util.concurrent.TimeUnit,int,java.lang.String,com.XXX.framework.annotations.TestNameParameter)(value(s): NANOSECONDS, 60, "60ns", TestNameParameter(customName=60 NANOSECONDS))

...
Test executes and results, etc.

On Jenkins:

[2023-06-06 21:02:29.270] [INFO] [TestClass] Creating TestClass for [ClassImpl class=com.XXX.framework.utils.TimeUtilsTest]
[2023-06-06 21:02:29.271] [INFO] Method public java.lang.Object[][] com.XXX.framework.utils.TimeUtilsTest.getTimeData() has a @Test annotation but also a return value: ignoring it. Use <suite allow-return-values="true"> to fix this
[2023-06-06 21:02:29.271] [INFO] [TestClass] Adding method TimeUtilsTest.testToFormattedString(java.util.concurrent.TimeUnit,int,java.lang.String,com.XXX.framework.annotations.TestNameParameter)[pri:0, instance:null] on TestClass class com.XXX.framework.utils.TimeUtilsTest

...

[2023-06-06 21:02:29.529] [INFO] ===== Test class
com.XXX.framework.utils.TimeUtilsTest
[2023-06-06 21:02:29.529] [INFO]     @Test TimeUtilsTest.testToFormattedString(java.util.concurrent.TimeUnit,int,java.lang.String,com.XXX.framework.annotations.TestNameParameter)[pri:0, instance:com.XXX.framework.utils.TimeUtilsTest@52f9a620]
[2023-06-06 21:02:29.529] [INFO] ======

I see no other references about the test class in the rest of the trace. Basically, all tests with a DataProvider don't run.

Maybe someone has a suggestion on what else to look for.

I've tried the following:

• Making sure the proper Test annotation is used (TestNG vs. JUnit
• Remove test groups or any other listener that could be interfering.
• Downgrade the TestNG version.
• Pray.

Here's the investigation on the TestNG side: https://github.com/testng-team/testng/issues/2924

Question in StackOverflow: https://stackoverflow.com/questions/76418882/why-is-testng-not-running-tests-with-dataprovider