[SUREFIRE-611] surefire exit code should explicitly distinguish between crashed and failed unit test(s)

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.11
• Component/s: Maven Surefire Plugin
• Assignee: Kristian Rosenvold
• Reporter: Tycho Lamerigts
• Created: 23/Mar/10 07:09
• Updated: 22/Nov/11 15:47
• Resolved: 22/Nov/11 15:47

Description

• If a unit test crashes, surefire skips all remaining tests, prints a message "[ERROR] There are test failures. ..." to the console, and does NOT print a test summary report.

• If one or more unit tests fail, maven completes remaining tests, prints a message "[ERROR] There are test failures. ..." to the console, and DOES print a test summary report.

The only distinction between crash and fail is the presence (or absence) of a summary report.

This is a problem for some tools (like TeamCity), because they typically don't pick up on the fact that many testcases were not run at all in case of a crashed unit test. Instead in case of a crashed test they simply report, for example, 100 tests succeeded, 0 failed. They do not report "50 tests skipped due to a crash after test #100", and it is quite hard for those tools to recognize this situation. If surefire would report a crash more explicitly, other tools can act accordingly.

I have attached a small maven project to reproduce the problem. By commenting and uncommenting the marked line in file ATest.java, you can reproduce both fail and crash scenarios. In the crash scenario, Maven will correctly report a failed build (unless you use the -Dmaven.test.failure.ignore flag, in which case it even won't do that), but it fails to indicate there was a crash which caused tests (in this case 3) to be skipped.