Details
-
New Feature
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
Pull Request 40:
https://github.com/apache/maven-surefire/pull/40
Add rerunFailingTestsCount option for maven surefire to rerun failing tests immediately after they fail.
When rerunFailingTestsCount is set to a value k larger than 0, a failing test will get re-run
up to k times until it passes. If a test passes in any of its reruns,
the build will be marked as successful and the test will count as a
flake (or flaky test). If it fails all those k times then it will
still be marked as a failed test.
In the console output all the flaky tests will be count as "Flakes:
N". The generated test report XML file is augmented with additional
information, while still being compatible with existing consumers
(such as Jenkins). A flaky test will have <flakyFailure> or/and
<flakyError> under its <testcase> element, to store all the flaky
runs' information (such as output, stackTrace). So existing consumers
will still consider it as a passing test, while potential future
consumers can parse those flaky runs information. A failing test will
still have <failure> or <error> under <testcase>, but all the
subsequent re-run information will be stored under <rerunFailure> or
<rerunError>. So existing consumers will still be able to see it's a
failed test and parse its failure information, and potential future
consumers will be able to get all the flaky runs.
It is implemented by keeping a map between test full class name and a
map between all its test methods and the list of runs. It also takes
into account Fork and Parallel and have them covered by integration
tests.
Currently only supports JUnit4.x