[SUREFIRE-1914] XML report omits method signature / display name of Junit 5 parameterized tests if testset reporter is configured to use phrased naming

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-M5
• Fix Version/s: 3.0.0-M6
• Component/s: JUnit 5.x support, xml generation
• Assignee: Tibor Digana
• Reporter: Andreas Pabst
• Created: 16/May/21 14:22
• Updated: 03/Oct/22 16:08
• Resolved: 13/Jul/21 21:47

Description

Description of the issue

Given a test class with two parameterized tests as follows:

    @ParameterizedTest
    @ValueSource(strings = {"a", "b"})
    void test1(String param) {
        // test code
    }

    @ParameterizedTest
    @ValueSource(strings = {"a", "b"})
    void test2(String param) {
        // test code
    }

and a surefire configuration that includes the following:

<statelessTestsetReporter implementation="org.apache.maven.plugin.surefire.extensions.junit5.JUnit5Xml30StatelessReporter">
    <version>3.0</version>
    <usePhrasedTestSuiteClassName>true</usePhrasedTestSuiteClassName>
    <usePhrasedTestCaseClassName>true</usePhrasedTestCaseClassName>
    <usePhrasedTestCaseMethodName>true</usePhrasedTestCaseMethodName>
</statelessTestsetReporter>

the XML report will look like this:

  <testcase name="[1] a" classname="mypackage.MyClass" time="0.001" />
  <testcase name="[2] b" classname="mypackage.MyClass" time="0.001" />
  <testcase name="[1] a" classname="mypackage.MyClass" time="0.001" />
  <testcase name="[2] b" classname="mypackage.MyClass" time="0.001" />

Note: The test method name/signature is not included in the name attribute.

I would expect something more like this:

  <testcase name="test1(String) [1] a" classname="mypackage.MyClass" time="0.001" />
  <testcase name="test1(String) [2] b" classname="mypackage.MyClass" time="0.001" />
  <testcase name="test2(String) [1] a" classname="mypackage.MyClass" time="0.001" />
  <testcase name="test2(String) [2] b" classname="mypackage.MyClass" time="0.001" />

Some context on why this is bad

Omitting the test method name makes it impossible to differentiate individual test cases in the XML report if there are multiple @ParameterizedTest-annotated test cases in the same class that use the same parameters.

Any software that parses the surefire XML reports will be misled into thinking these are multiple executions of the same test.

Variant with @DisplayName usage

There is a variant of this issue: If the @ParameterizedTest has a @DisplayName annotation, whatever was chosen as a display name is not included in the XML report either - unless it is explicitly referenced in the name attribute of the @ParameterizedTest like so: @ParameterizedTest(name = "{displayName} ...").

Solution ideas

This issue has already been brought up during the discussion of SUREFIRE-1546.

In that discussion srdo and marcphilipp have described an approach how CONTAINER-type TestIdentifiers could be handled properly, but that particular problem seems to have been deferred and has seemingly not been picked up again since.

Workaround

A workaround is to explicitly reference the displayName or include some other unique string in the name attribute of each @ParameterizedTest as mentioned above. In projects with a large legacy testbase that might not be feasible though, so this issue might prevent the adoption of the feature introduced in SUREFIRE-1546.

I'm submitting this as a bug because to me the current behaviour of surefire is at least unexpected, but it could also be seen as a request for improvement.

Issue Links

• is related to: SUREFIRE-2117 XML report omits package and outer class name for tests in @Nested inner class if testset reporter is configured to use phrased naming
• links to: GitHub Pull Request #351
• links to: GitHub Pull Request #352
• links to: GitHub Pull Request #352