[SUREFIRE-1556] Test XML file is not valid when rerun "fails" with an assumption

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.21.0, 2.22.0
• Fix Version/s: 2.22.3, 3.0.0-M6
• Component/s: Maven Surefire Plugin
• Assignee: Tibor Digana
• Reporter: Michael Weisgrab
• Created: 21/Aug/18 08:39
• Updated: 02/Feb/22 23:47
• Resolved: 02/Feb/22 20:26

Description

We are running UI tests and are doing one retry if there is a bug.

When a test fails with an Exception on the first try and the second try is skipped by an assumption violation the resulting test report xml file is not valid (not well formatted xml file).

When the test is skipped on the first try or both runs fail with an exception everything is ok.

This is a testcase which is retried and a valid xml is generated.

 

<testcase name="xxx" classname="xxx" time="47.905">
    <error message="message" type="org.openqa.selenium.TimeoutException">
        ...
    </error>
    <system-out>
        ...
    </system-out>
    <system-err>
        ...
    </system-err>
    <rerunError message="message" type="java.lang.IndexOutOfBoundsException">
        ...
        <system-out>
            ...
        </system-out>
        <system-err>
            ...
        </system-err>
    </rerunError>
</testcase>

 

 

Here is a generated invalid xml. The rerunError tag is missing. The retry "error" is an assumption violation so the test should be marked as skipped with message of the assumptionviolation.

<testcase name="xxx" classname="xxx" time="54.153">
  <error message="message" type="org.openqa.selenium.TimeoutException">

  </error>
  <system-out>
    ...
  </system-out>
  <system-err>
    ...
  </system-err>
  < message="There should be a list of ads!">
    <system-out>
      ...
    </system-out>
    <system-err>
      ...
    </system-err>
  </>
</testcase>

I have attached the following files:

• TEST-asdf.SearchTest_assumption_valid.xml (assumption violation in the first test run)
• TEST-asdf.SearchTest_rerun_fail_valid.xml (both runs throw exception)
• TEST-asdf.SearchTest_rerun_assumption_invalid.xml (first run exception - second run assumption violation)

Issue Links

• is related to: SUREFIRE-1986 New element "rerunSkipped" in XSD of XML Test Report
• links to: GitHub Pull Request #11
• links to: GitHub Pull Request #450