[SUREFIRE-1127] Failsafe project does not fail in verify phase when a test case object errors during initialization

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.18
• Fix Version/s: 2.19
• Component/s: Maven Failsafe Plugin
• Environment: JDK 1.8.0_05 on Windows 7 used to recreate
• Assignee: Tibor Digana
• Reporter: Bret Goldsmith
• Created: 21/Dec/14 16:07
• Updated: 23/May/16 09:49
• Resolved: 15/Jun/15 22:19

Description

Sample project attached.

When running a "mvn verify" - the attached project will succeed, even though there is an integration test that should obviously fail.

The integration test case does not initialize correctly (it will throw a null pointer exception in one of its object initializers), which should be a test failure and therefore a project failure.

I've traced this down to the fact that the generated failsafe-summary.xml file contains an empty failureMessage element:

<?xml version="1.0" encoding="UTF-8"?>
<failsafe-summary result="254" timeout="false">
  <completed>0</completed>
  <errors>0</errors>
  <failures>0</failures>
  <skipped>0</skipped>
  <failureMessage></failureMessage>
</failsafe-summary>

And so the failsafe verifier doesn't consider the project a failure. The message seems to be blank because when the failsafe plugin is writing the results, it uses the org.apache.maven.plugins.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked method to write out the first forked exception, which then calls org.apache.maven.surefire.suite.RunResult.failure() method, which then calls getStackTrace() on the exception.

The getStackTrace() method looks like this:

    private static String getStackTrace( Exception e )
    {
        if ( e == null )
        {
            return null;
        }
        ByteArrayOutputStream out = new ByteArrayOutputStream();
        PrintWriter pw = new PrintWriter( out );
        e.printStackTrace( pw );
        return new String( out.toByteArray() );
    }

And it specifically is doing a toBytearray on the output stream without flushing or closing the printwriter first. At least with my JVM 1.8.0_05 version, this seems to be a problem as the printwriter still maintains the trace description in a buffer.

When I put a breakpoint on this and specifically flush or close the printwriter before the toByteArray, the project fails as expected.

I'm not sure why this is failsafe specific, but a surefire version test case (also included in the attachment project) fails appropriately.

I think a simple patch is to update RunResult.java to close the printwriter before getting the resulting stacktrace string. i.e.

    private static String getStackTrace( Exception e )
    {
        if ( e == null )
        {
            return null;
        }
        ByteArrayOutputStream out = new ByteArrayOutputStream();
        PrintWriter pw = new PrintWriter( out );
        try {
            e.printStackTrace( pw );
        } finally {
            pw.close();
        }
        return new String( out.toByteArray() );
    }

Issue Links

• is related to: FLINK-3909 Maven Failsafe plugin may report SUCCESS on failed tests