Uploaded image for project: 'Maven Surefire'
  1. Maven Surefire
  2. SUREFIRE-1556

Test XML file is not valid when rerun "fails" with an assumption

    XMLWordPrintableJSON

Details

    Description

      We are running UI tests and are doing one retry if there is a bug.

      When a test fails with an Exception on the first try and the second try is skipped by an assumption violation the resulting test report xml file is not valid (not well formatted xml file).

      When the test is skipped on the first try or both runs fail with an exception everything is ok.

      This is a testcase which is retried and a valid xml is generated.

       

      <testcase name="xxx" classname="xxx" time="47.905">
          <error message="message" type="org.openqa.selenium.TimeoutException">
              ...
          </error>
          <system-out>
              ...
          </system-out>
          <system-err>
              ...
          </system-err>
          <rerunError message="message" type="java.lang.IndexOutOfBoundsException">
              ...
              <system-out>
                  ...
              </system-out>
              <system-err>
                  ...
              </system-err>
          </rerunError>
      </testcase>

       

       

      Here is a generated invalid xml. The rerunError tag is missing. The retry "error" is an assumption violation so the test should be marked as skipped with message of the assumptionviolation.

      <testcase name="xxx" classname="xxx" time="54.153">
        <error message="message" type="org.openqa.selenium.TimeoutException">
      
        </error>
        <system-out>
          ...
        </system-out>
        <system-err>
          ...
        </system-err>
        < message="There should be a list of ads!">
          <system-out>
            ...
          </system-out>
          <system-err>
            ...
          </system-err>
        </>
      </testcase>
      

      I have attached the following files:

      • TEST-asdf.SearchTest_assumption_valid.xml (assumption violation in the first test run)
      • TEST-asdf.SearchTest_rerun_fail_valid.xml (both runs throw exception)
      • TEST-asdf.SearchTest_rerun_assumption_invalid.xml (first run exception - second run assumption violation)

      Attachments

        1. TEST-asdf.SearchTest_assumption_valid.xml
          0.5 kB
          Michael Weisgrab
        2. TEST-asdf.SearchTest_rerun_assumption_invalid.xml
          0.7 kB
          Michael Weisgrab
        3. TEST-asdf.SearchTest_rerun_fail_valid.xml
          0.9 kB
          Michael Weisgrab

        Issue Links

          Activity

            People

              tibordigana Tibor Digana
              Micky002 Michael Weisgrab
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 10m
                  10m