Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Duplicate
-
2.22.1, 2.22.2, 3.0.0-M4
-
None
-
None
Description
Running JUnit5 tests using maven-surefire-plugin with <reuseForks>true</reuseForks> e.g. mvn -DforkCount=8
Results: number of tests in each fork is significantly unbalanced, some forks do not get tests at all..
The end results is that on CI server the total test execution will take as long as longest fork and only 50% of the available resources are used.
In our case we observed that our 3000 integration tests after migration to junit5 take 30 minutes insteads of 18 minutes as it was with junit 4.
The balance is restored when running with reuseForks=false but this will cause additional overhead for complex integrations tests
e.g. tests requiring additional setup , classpath scanning and so on..
The results are similar with maven-surefire-plugin: 2.22.1, 2.22.2, 3.0.0-M4
Maven 3.6.3 , openjdk version "1.8.0_242" on Linux
BTW If junit-vintage-engine is used the same results observed for JUnit3 and JUnit4 tests.
Consider example of 100 tests (10 classes 10 tests in each)
https://github.com/skarzhevskyy/range-junit/tree/master/range-junit5-surefire-fork
When running mvn -DforkCount=8 with extension that prints the number of tests executed in each fork getting following:
Tests completed JVM Fork 8; total tests: 0; classes: 0
Tests completed JVM Fork 4; total tests: 0; classes: 0
Tests completed JVM Fork 1; total tests: 0; classes: 0
Tests completed JVM Fork 6; total tests: 0; classes: 0
Tests completed JVM Fork 3; total tests: 0; classes: 0
Tests completed JVM Fork 2; total tests: 0; classes: 0
Tests completed JVM Fork 7; total tests: 30; classes: 3
Tests completed JVM Fork 5; total tests: 70; classes: 7
mvn -DforkCount=4
Tests completed JVM Fork 2; total tests: 0; classes: 0
Tests completed JVM Fork 3; total tests: 10; classes: 1
Tests completed JVM Fork 4; total tests: 40; classes: 4
Tests completed JVM Fork 1; total tests: 50; classes: 5
The distribution is also random, results may vary between maven runs:
mvn -DforkCount=4
Tests completed JVM Fork 4; total tests: 0; classes: 0
Tests completed JVM Fork 2; total tests: 10; classes: 1
Tests completed JVM Fork 1; total tests: 30; classes: 3
Tests completed JVM Fork 3; total tests: 60; classes: 6
Attachments
Issue Links
- duplicates
-
SUREFIRE-1725 Surefire in JUnit Vintage mode distributes tests very unevenly between forks, causing poor parallelism
- Closed