[SUREFIRE-1794] @TestFactory method succeeds when throwing an exception with Maven Failsafe plugin

• Type: Task
• Status: Closed
• Priority: Major
• Resolution: Cannot Reproduce
• Assignee: Tibor Digana
• Reporter: Zhenxu Ke
• Created: 20/May/20 14:51
• Updated: 21/May/20 08:32
• Resolved: 21/May/20 08:32

Description

Hi, this is an issue transferred from https://github.com/junit-team/junit5/issues/2303

 

Steps to reproduce

Here is a reproducible test:

 

@TestFactory
    List<DynamicNode> tests() {
        int a = 1 / 0;  // intentionally throws
        return Arrays.asList(
            DynamicContainer.dynamicContainer(
                "dynamic container 1",
                Stream.of(
                    DynamicTest.dynamicTest("dynamic test 1", () -> fail("")),
                    DynamicTest.dynamicTest("dynamic test 2", () -> fail(""))
                )
            ),
            DynamicContainer.dynamicContainer(
                "dynamic container 2",
                Stream.of(
                    DynamicTest.dynamicTest("dynamic test 3", () -> fail("")),
                    DynamicTest.dynamicTest("dynamic test 4", () -> fail(""))
                )
            )
        );
    }

 

the test above succeeded if `int a = 1 / 0` is present, and failed as expected if I remove it, showing 4 tests executed and all of them failed.

Context

• Used versions (Jupiter/Vintage/Platform):

 

<dependency>
  <groupId>org.junit.jupiter</groupId>
  <artifactId>junit-jupiter</artifactId>
  <version>5.6.0</version>
</dependency>

• Build Tool/IDE:

maven-failsafe-plugin, with all necessary dependencies configured
 

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-failsafe-plugin</artifactId>
                <version>3.0.0-M4</version>
                <dependencies>
                    <!-- not important, thus omitted -->
                </dependencies>
            </plugin>

Deliverables

I'm expecting the tests to be failed if the TestFactory method throws