[SUREFIRE-1970] rerunFailingTestsCount option marks some spock tests as false positive

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0-M4, 3.0.0-M5
• Component/s: Maven Surefire Plugin
• Labels: groovy maven rerun spock surefire

Language

groovy

• Assignee: Unassigned
• Reporter: Casey Jones
• Created: 23/Dec/21 03:12
• Updated: 23/Dec/21 03:12

Description

I have a project which uses Spock Framework for tests. There is a single test which intentionally fails, no matter how many times it was ran:

class FalsePositiveTest extends Specification {
    def "test"() {
        expect:
        1 == number
        where:
        number << [2, 3]    
    }
} 

When test is executed with mvn test command, the build fails, as expected. 

However, when using surefire's rerunFailingTestsCount option, the test in question is executed two times, as expected, but then it is marked as flaky, and the build is successful:

mvn test -Dsurefire.rerunFailingTestsCount=1 
...
Warning:  Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
126[INFO] 
127[INFO] ------------------------------------------------------------------------
128[INFO] BUILD SUCCESS
129[INFO] ------------------------------------------------------------------------

Please check the links below for a full build log.

This issue happens only when using Spock's data-driven tests with a where block and surefire's rerunFailingTestsCount option together. It seems like there is something wrong in the way surefire detects if test has failed or passed.

I asked this question on StackOverflow and Spock's maintainer suggested to raise an issue in surefire's bugtracker.

Here is a simple github project which reproduces the issue: https://github.com/streetturtle/FalsePositiveTest

Please check build logs of a github action here: https://github.com/streetturtle/FalsePositiveTest/runs/4613803891?check_suite_focus=true#step:4:128