Uploaded image for project: 'Maven Surefire'
  1. Maven Surefire
  2. SUREFIRE-1627

Surefire report contains wrong totals on number of tests and errors when used with rerunFailingTestsCount flag

Agile BoardAttach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 2.19.1
    • None
    • None

    Description

      Steps:

      1. Make sure you have at least one Test class with at least one test method
      2. Make this test method be constantly failing throwing an `AssertionError`
      3. Run the test via maven using rerun option: `mvn clean test -Dsurefire.rerunFailingTestsCount=5`
      4. Compare build output and xml report generated by Surefire.

      Issue: the report ` target/surefire-reports/TEST-mainTest.xml` contains wrong counts on total number of tests and failures (in the top level element):

      <testsuite xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report.xsd" name="mainTest" time="0.05" tests="6" errors="0" skipped="0" failures="6">
      
      

      Please note that there is only 1 element testcase in the body (versus 6 in the totals), and only one element failure as well (versus 6 in the totals). The report is considering each re-run as a separate test and each rerunFailure as a failure count. That leads to inconsistencies when parsing the report since there is mismatch between totals and actual number of tree elements. 

      Also it contradicts general reporting logic one can see in the build output:

      Results :
      
      Failed tests: 
      mainTest.failingTest(mainTest)
        Run 1: mainTest.failingTest:9 Failing test
        Run 2: mainTest.failingTest:9 Failing test
        Run 3: mainTest.failingTest:9 Failing test
        Run 4: mainTest.failingTest:9 Failing test
        Run 5: mainTest.failingTest:9 Failing test
        Run 6: mainTest.failingTest:9 Failing test
      
      
      Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
      
      [INFO] ------------------------------------------------------------------------
      [INFO] BUILD FAILURE
      [INFO] ------------------------------------------------------------------------
      
      

       
      To sum up, reporting counts not just  rerunFailure, but also flakyFailure as a new failure, and each rerun as a new test for totals counts.

      I see the same issue with Surefire versions 2.22.1 and 3.0.0-M3 as well.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            alex_filatau Alex Filatau

            Dates

              Created:
              Updated:

              Slack

                Issue deployment