[SUREFIRE-2039] Skipped test classes are getting into the non-skipped test classes reports

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 3.0.0-M5
• Assignee: Unassigned
• Reporter: Semyon Danilov
• Created: 20/Mar/22 12:37
• Updated: 08/Mar/23 23:44
• Resolved: 08/Mar/23 23:42

Description

The setup: maven-surefire-plugin 3.0.0-M5 and junit 5.8.2.

Consider a simple scenario, two test classes, ATest and BTest, one Disabled and one not respectively, each has three test methods. After running mvn surefire:test the report will indicate that 4 tests were run and one of them was skipped:

[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.example.test.BTest
[WARNING] Tests run: 4, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.062 s - in org.example.test.BTest
[INFO] 
[INFO] Results:
[INFO] 
[WARNING] Tests run: 4, Failures: 0, Errors: 0, Skipped: 1

And if we enable ATest and disable BTest, then the report will only mention running three tests, none skipped:

[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.example.test.ATest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.079 s - in org.example.test.ATest
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0

This behavior is the result of the way how org.apache.maven.plugin.surefire.report.TestSetRunListener handles skipped tests. It adds them to the detailsForThis and only prints after the test set is finished which happens when a non-disabled test set is finished thus adding skipped test results to the statistics of a non-skipped test set. And, of course, if the disabled test is ran after the non-disabled test, it won't be reported at all.

The reproducer can be found here: https://github.com/SammyVimesFiledIssues/SUREFIRE-2039

Issue Links

• duplicates: SUREFIRE-2032 When declare the @Disabled annotation at the class level, skip displayed in other test class
• links to: GitHub Pull Request #493
• links to: The reproducer