[SUREFIRE-2087] rerunFailingTestsCount incorrectly marks failed parametrized test as pass

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 3.0.0-M4
• Assignee: Unassigned
• Reporter: Ramanan Rajendran
• Created: 19/May/22 16:03
• Updated: 21/Feb/23 09:30
• Resolved: 16/Feb/23 18:53

Description

rerunFailingTestsCount is not working as expected with junit5 `@ParameterizedTest` 

If 1st iteration of the test passes and the 2nd one fails, it still marks the test as flaky passed. 

Eg: In the following run, iteration 1 with input A passed but iteration 2 with input B failed. Iteration 2 was rerun 2 times and failed in all the attempts. Instead of failing TestA, it's marked as flaky giving an overall pass for the test suite. I believe it's using the Run 1 PASS as a gate to mark the whole test as a pass. 

 
[INFO] Run 1: PASS
[ERROR] Run 2: TestA.iteration2:101 1 expectation failed.
Expected status code <400> doesn't match actual status code <401>.
[ERROR] Run 3: TestA.iteration2:101 1 expectation failed.
Expected status code <400> doesn't match actual status code <401>.
[ERROR] Run 4: TestA.iteration2:101 1 expectation failed.
<400> doesn't match actual status code <401>.
 
Tests run: 23, Failures: 0, Errors: 0, Skipped: 2, Flakes: 1

Issue Links

• duplicates: SUREFIRE-2065 Test Reports Inconsistencies with Parameterized and junit4