[SUREFIRE-1799] Exception thrown in @MethodSource method results in "no tests"

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 3.0.0-M4
• Component/s: JUnit 5.x support
• Assignee: Tibor Digana
• Reporter: Jens Reimann
• Created: 15/Jun/20 07:20
• Updated: 15/Jun/20 21:56
• Resolved: 15/Jun/20 21:52

Description

Running the following code:

import java.util.List;

import org.junit.jupiter.api.TestInstance;
import org.junit.jupiter.api.TestInstance.Lifecycle;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.MethodSource;

@TestInstance(Lifecycle.PER_CLASS)
public class MethodSourceErrorTest {

    public List<String> getValues() {
        throw new RuntimeException("Foo"); // fails
        // return Arrays.asList("a"); // works
    }

    @ParameterizedTest
    @MethodSource("getValues")
    public void test1(String value) {
        System.out.println(value);
    }

}

Results in:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running foo.MethodSourceErrorTest
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.018 s - in foo.MethodSourceErrorTest
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------

My expectation would be that:

• JUnit reports an error for the tests using the method source
• The build fails

Additional information:

• It fails the same way, being a static method

Used versions (Jupiter/Vintage/Platform): 5.6.2, 1.6.2

I raised an issue with JUnit 5 (https://github.com/junit-team/junit5/issues/2319), however they claim that it is an issue with the tool running the test (Maven Surefire in this case).

And indeed the TestExecutionListener receives an exception:

public class MyTestExecutionListener implements TestExecutionListener {

    public MyTestExecutionListener() {
        System.out.println("Here");
    }

    @Override
    public void executionFinished(TestIdentifier testIdentifier, TestExecutionResult testExecutionResult) {
        System.out.format("finished - id: %s, result: %s%n", testIdentifier, testExecutionResult);
    }
}

Shows the exception as:

[INFO] --- maven-surefire-plugin:3.0.0-M4:test (default-test) @ bar ---
[WARNING] useSystemClassloader setting has no effect when not forking
Here
[INFO] Running foo.MethodSourceErrorTest
finished - id: TestIdentifier [uniqueId = '[engine:junit-jupiter]/[class:foo.MethodSourceErrorTest]/[test-template:test1(java.lang.String)]', parentId = '[engine:junit-jupiter]/[class:foo.MethodSourceErrorTest]', displayName = 'test1(String)', legacyReportingName = 'test1(String)', source = MethodSource [className = 'foo.MethodSourceErrorTest', methodName = 'test1', methodParameterTypes = 'java.lang.String'], tags = [], type = CONTAINER], result: TestExecutionResult [status = FAILED, throwable = java.lang.RuntimeException: Foo]
finished - id: TestIdentifier [uniqueId = '[engine:junit-jupiter]/[class:foo.MethodSourceErrorTest]', parentId = '[engine:junit-jupiter]', displayName = 'MethodSourceErrorTest', legacyReportingName = 'foo.MethodSourceErrorTest', source = ClassSource [className = 'foo.MethodSourceErrorTest', filePosition = null], tags = [], type = CONTAINER], result: TestExecutionResult [status = SUCCESSFUL, throwable = null]
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.013 s - in foo.MethodSourceErrorTest
finished - id: TestIdentifier [uniqueId = '[engine:junit-jupiter]', parentId = null, displayName = 'JUnit Jupiter', legacyReportingName = 'JUnit Jupiter', source = null, tags = [], type = CONTAINER], result: TestExecutionResult [status = SUCCESSFUL, throwable = null]
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

Issue Links

• duplicates: SUREFIRE-1741 Exceptions in parameterized test sources are ignored