Sample project attached.
When running a "mvn verify" - the attached project will succeed, even though there is an integration test that should obviously fail.
The integration test case does not initialize correctly (it will throw a null pointer exception in one of its object initializers), which should be a test failure and therefore a project failure.
I've traced this down to the fact that the generated failsafe-summary.xml file contains an empty failureMessage element:
And so the failsafe verifier doesn't consider the project a failure. The message seems to be blank because when the failsafe plugin is writing the results, it uses the org.apache.maven.plugins.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked method to write out the first forked exception, which then calls org.apache.maven.surefire.suite.RunResult.failure() method, which then calls getStackTrace() on the exception.
The getStackTrace() method looks like this:
And it specifically is doing a toBytearray on the output stream without flushing or closing the printwriter first. At least with my JVM 1.8.0_05 version, this seems to be a problem as the printwriter still maintains the trace description in a buffer.
When I put a breakpoint on this and specifically flush or close the printwriter before the toByteArray, the project fails as expected.
I'm not sure why this is failsafe specific, but a surefire version test case (also included in the attachment project) fails appropriately.
I think a simple patch is to update RunResult.java to close the printwriter before getting the resulting stacktrace string. i.e.