The Surefire plugin currently covers two use cases with its `rerunFailingTestCount` option:
1. rerunFailingTestCount=0: The build will fail immediately, but I don't know if it was a real failure or a flake. This option is best for healthy code bases with very little or no flakiness.
2. rerunFailingTestCount>0: I will know when a test is flaky, but the build will be successful, so there is no pressure to fix the flakes. This helps when the situation is really bad and I just want to get some green build again to boost team morale
I'd like to support a third use case:
3. rerunFailingTestCount>0 and a new option failOnFlakeCount > 0: The build will fail (so there is actually pressure to deal with flakiness) and I can tell the difference between real failures and flakes. I think this would be the best option for projects with a medium amount of flaky tests and a team that wants to take care of them.
Edit: To clarify, this is about failing the whole test set when there is flakiness, not individual tests.