[SUREFIRE-779] Surefire reports wrong number of failed tests when using JUnit's ErrorCollector rule

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 2.10
• Component/s: Maven Surefire Plugin
• Assignee: Unassigned
• Reporter: Lieven Doclo
• Created: 21/Oct/11 07:59
• Updated: 31/May/24 20:30

Description

When running a test that contains an ErrorCollector, test failures are not correct when an errorcollector contains failures. For example:

public class ExampleTest {
    @Rule
    public ErrorCollector errorCollector = new ErrorCollector();

    @Test
    public void testWithErrorCollector() {
        errorCollector.checkThat(true, is(false));
        errorCollector.checkThat(3, is(4));
    }
}

Surefire will reports 2 tests run (of which all failed). However, when you fix the same test so that no checks fail, Surefire will report 1 test run.

This is incorrect, as this will result in false information in systems like Hudson or Sonar, the number of testcase should not fluctuate. Surefire should only report 1 test run, as that reflects the actual situation.

Issue Links

• Blocked: SUREFIRE-841 Incorrect Test Run Count
• is duplicated by: SUREFIRE-1172 Incompatibility with MultipleFailureException
• relates to: SUREFIRE-841 Incorrect Test Run Count