[SUREFIRE-1688] Junit Platform test with failing @BeforeAll does not fail the build

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Component/s: JUnit 5.x support, Maven Surefire Plugin
• Assignee: Tibor Digana
• Reporter: Guy Brand
• Created: 20/Aug/19 14:24
• Updated: 21/Apr/20 11:21
• Resolved: 10/Feb/20 21:48

Description

When using Surefire 3.0.0-SNAPSHOT and JUnit Platform 5.4.2 the following test does not fail the Maven build.

import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Assertions;
public class Tests {

    @BeforeAll
    public static void beforeAll() { Assertions.fail("BeforeAll fail"); }

    @Test
    public void test() { Assertions.fail("Test fail"); }
}

And its output:

[INFO] --- maven-surefire-plugin:3.0.0-SNAPSHOT:test (default-test) @ top-level-project ---
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running Tests
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.015 s - in Tests
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------

When removing the @BeforeAll annotation/method it fails as expected.

3.0.0-M3 version of Surefire works as expected:

[INFO] --- maven-surefire-plugin:3.0.0-M3:test (default-test) @ top-level-project ---
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running Tests
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.008 s <<< FAILURE! - in Tests
[ERROR] Tests  Time elapsed: 0.007 s  <<< FAILURE!
org.opentest4j.AssertionFailedError: BeforeAll fail
	at Tests.beforeAll(Tests.java:7)

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   Tests.beforeAll:7 BeforeAll fail
[INFO] 
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  2.373 s
[INFO] Finished at: 2019-08-20T16:20:16+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M3:test (default-test) on project top-level-project: There are test failures.

For all the tests Maven 3.6.1 has been used.

Issue Links

• duplicates: SUREFIRE-1741 Exceptions in parameterized test sources are ignored
• is duplicated by: SUREFIRE-1730 Exception in BeforeAll and AfterAll is not handled
• is duplicated by: SUREFIRE-1750 Wrong resolution of JUnit5 Provider artifacts
• links to: GitHub Pull Request #255
• links to: GitHub Pull Request #258