[SUREFIRE-1741] Exceptions in parameterized test sources are ignored

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-M4
• Fix Version/s: 3.0.0-M5
• Component/s: JUnit 5.x support, Maven Failsafe Plugin, Maven Surefire Plugin
• Assignee: Tibor Digana
• Reporter: Albert Johnston
• Created: 27/Dec/19 22:58
• Updated: 21/Aug/20 11:22
• Resolved: 01/Feb/20 01:20

Description

Associated versions on my end at time of testing:

• JUnit Jupiter 5.5.1
• JDK 11.0.5
• Maven 3.6.1

 

If the following code is run, either for Surefire or Failsafe, only two of the test methods are logged as having been detected at all. The third one is completely ignored as if it didn't exist.

public class ExampleTest {

  @Test
  public void shouldRunAndPassAsExpected() {
  }

  public static Stream<Arguments> nothing() {
    return Stream.of(Arguments.arguments());
  }

  @ParameterizedTest
  @MethodSource("nothing")
  public void shouldRunAndFailAsExpected() {
    Assertions.fail();
  }

  static Stream<Arguments> throwException() {
    if (true) throw new RuntimeException("Stop");
    return Stream.of(Arguments.arguments());
  }

  @ParameterizedTest
  @MethodSource("throwException")
  void willNotBeLoggedAsFailingDespiteNotRunning() {
    Assertions.fail();
  }
}

This is particularly obvious if the first two test methods are commented out, as running mvn test will log that 0 tests were run.

 

In case it helps, here's the output I'm seeing when running it with the first two tests commented out:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running ExampleIT
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.02 s - in ExampleIT
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

 

Issue Links

• is duplicated by: SUREFIRE-1688 Junit Platform test with failing @BeforeAll does not fail the build
• is duplicated by: SUREFIRE-1727 Failures during test template creation are ignored
• is duplicated by: SUREFIRE-1799 Exception thrown in @MethodSource method results in "no tests"
• is related to: SUREFIRE-1739 Regression - Failsafe with JUnit 5: @BeforeAll failure incorrectly succeeds
• links to: GitHub Pull Request #267
• links to: GitHub Pull Request #269