Uploaded image for project: 'Maven Surefire'
  1. Maven Surefire
  2. SUREFIRE-1556

Test XML file is not valid when rerun "fails" with an assumption

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.21.0, 2.22.0
    • Fix Version/s: 3.0.0-M6
    • Component/s: Maven Surefire Plugin
    • Labels:
      None

      Description

      We are running UI tests and are doing one retry if there is a bug.

      When a test fails with an Exception on the first try and the second try is skipped by an assumption violation the resulting test report xml file is not valid (not well formatted xml file).

      When the test is skipped on the first try or both runs fail with an exception everything is ok.

      This is a testcase which is retried and a valid xml is generated.

       

      <testcase name="xxx" classname="xxx" time="47.905">
          <error message="message" type="org.openqa.selenium.TimeoutException">
              ...
          </error>
          <system-out>
              ...
          </system-out>
          <system-err>
              ...
          </system-err>
          <rerunError message="message" type="java.lang.IndexOutOfBoundsException">
              ...
              <system-out>
                  ...
              </system-out>
              <system-err>
                  ...
              </system-err>
          </rerunError>
      </testcase>

       

       

      Here is a generated invalid xml. The rerunError tag is missing. The retry "error" is an assumption violation so the test should be marked as skipped with message of the assumptionviolation.

      <testcase name="xxx" classname="xxx" time="54.153">
        <error message="message" type="org.openqa.selenium.TimeoutException">
      
        </error>
        <system-out>
          ...
        </system-out>
        <system-err>
          ...
        </system-err>
        < message="There should be a list of ads!">
          <system-out>
            ...
          </system-out>
          <system-err>
            ...
          </system-err>
        </>
      </testcase>
      

      I have attached the following files:

      • TEST-asdf.SearchTest_assumption_valid.xml (assumption violation in the first test run)
      • TEST-asdf.SearchTest_rerun_fail_valid.xml (both runs throw exception)
      • TEST-asdf.SearchTest_rerun_assumption_invalid.xml (first run exception - second run assumption violation)

        Attachments

        1. TEST-asdf.SearchTest_assumption_valid.xml
          0.5 kB
          Michael Weisgrab
        2. TEST-asdf.SearchTest_rerun_assumption_invalid.xml
          0.7 kB
          Michael Weisgrab
        3. TEST-asdf.SearchTest_rerun_fail_valid.xml
          0.9 kB
          Michael Weisgrab

          Issue Links

            Activity

              People

              • Assignee:
                tibordigana Tibor Digana
                Reporter:
                Micky002 Michael Weisgrab
              • Votes:
                1 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 10m
                  10m