[SUREFIRE-1963] Detecting multiple test-frameworks

• Type: New Feature
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Assignee: Slawomir Jaranowski
• Reporter: Slawomir Jaranowski
• Created: 21/Nov/21 17:32
• Updated: 20/Mar/22 19:00

Description

When multiple test frameworks are present on project classpath surefire chooses one framework for running test.

During automatically provider discover some of type test can be skipped without any knowledge and information for developer.

It is danger situation than developer can add second test framework to project and in the result some of test will be skipped silently.

We can force usage one or more needed providers by adding those to plugin dependency and we can have control on executing test.

Proposition

• add options multipleFrameworks empty (not set) by default, with possible value warn or fail

Scenario 1 - default

when

• multipleFrameworks = empty - not set
• many tests framework on classpath
• no providers on plugin dependency

then

• current behavior - choose first provider without warning about skipped test

Scenario 2

when

• multipleFrameworks = warn
• many tests framework on classpath
• no providers on plugin dependency

then

• print warn about multiple providers

Scenario 3

when

• multipleFrameworks = fail
• many tests framework on classpath
• no providers on plugin dependency

then

• print error about multiple providers and break build

Scenario 4

when

• many tests framework on classpath
• selected providers on plugin dependency

then

• ignore multipleFrameworks
• use providers from dependency list, no warning

Issue Links

• is related to: SUREFIRE-1911 Groovy-based JUnit 4 tests no longer run w/ 5 in classpath
• is related to: SUREFIRE-1527 Surefire does not discover junit5 test case when testng is in the classpath
• is related to: MINVOKER-285 Consider of using groovy-all once again
• relates to: SUREFIRE-2033 Regression: 3.0.0-M5 misidentifies JUnit 5 configuration as JUnit 4
• relates to: SUREFIRE-1458 When multiple surefire providers are available through the maven plugin dependency test are executed N times
• relates to: SUREFIRE-1527 Surefire does not discover junit5 test case when testng is in the classpath
• links to: GitHub Pull Request #399