Uploaded image for project: 'Maven Surefire'
  1. Maven Surefire
  2. SUREFIRE-1878

Add failOnFlakeCount option

    XMLWordPrintableJSON

    Details

      Description

      The Surefire plugin currently covers two use cases with its `rerunFailingTestCount` option:

      1. rerunFailingTestCount=0: The build will fail immediately, but I don't know if it was a real failure or a flake. This option is best for healthy code bases with very little or no flakiness.
      2. rerunFailingTestCount>0: I will know when a test is flaky, but the build will be successful, so there is no pressure to fix the flakes. This helps when the situation is really bad and I just want to get some green build again to boost team morale

      I'd like to support a third use case:

      3. rerunFailingTestCount>0 and a new option failOnFlakeCount > 0: The build will fail (so there is actually pressure to deal with flakiness) and I can tell the difference between real failures and flakes. I think this would be the best option for projects with a medium amount of flaky tests and a team that wants to take care of them.

      Edit: To clarify, this is about failing the whole test set when there is flakiness, not individual tests.

        Attachments

          Activity

            People

            • Assignee:
              tibordigana Tibor Digana
              Reporter:
              oehme Stefan Oehme
            • Votes:
              2 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: