[SUREFIRE-2274] failOnFlakeCount does not work for failsafe

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.5.0
• Component/s: Maven Failsafe Plugin
• Assignee: Unassigned
• Reporter: Daniel Fesenmeyer
• Created: 01/Oct/24 15:34
• Updated: 01/Oct/24 15:34

Description

The property failOnFlakeCount seems not to work for the failsafe plugin. At least not as it is documented here: https://maven.apache.org/surefire/maven-failsafe-plugin/verify-mojo.html

Expected behavior

When using the failsafe configuration property failOnFlakeCount, I'd expect that the build fails in case there are more than the specified count of flakes.

Actual behavior

The build succeeds, even when there are more than the specified count of flakes. On the console, and the TEST-***.xml report files, flakes are correctly reported. However, the report failsafe-summary.xml, does not contain a flakes property, and thus the verify step fails.

How to reproduce

I've created a reproducer here: https://github.com/danielFesenmeyer/failsafe-flakecount-reproducer

Just run the integration tests for this simple maven project:
mvn verify

The build shows the following output, which is correct, except for the information that the build succeeded (and status code 0):

...
[INFO] Running de.fesenmeyer.reproducer.ReproducerIT
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 s -- in de.fesenmeyer.reproducer.ReproducerIT
[INFO] 
[INFO] Results:
[INFO] 
[WARNING] Flakes: 
[WARNING] de.fesenmeyer.reproducer.ReproducerIT.test1
[ERROR]   Run 1: ReproducerIT.test1:12 » IllegalState Try 0 failed
[ERROR]   Run 2: ReproducerIT.test1:12 » IllegalState Try 1 failed
[INFO]   Run 3: PASS
[INFO] 
[WARNING] de.fesenmeyer.reproducer.ReproducerIT.test2
[ERROR]   Run 1: ReproducerIT.test2:17 » IllegalState Try 0 failed
[ERROR]   Run 2: ReproducerIT.test2:17 » IllegalState Try 1 failed
[INFO]   Run 3: PASS
[INFO] 
[INFO] 
[WARNING] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Flakes: 2
[INFO] 
[INFO] 
[INFO] --- failsafe:3.5.0:verify (default) @ failsafe-flakecount-reproducer ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
...
 

Hint

Probably the Feature currently works for Surefire, it seems like it has been only tested with Surefire.