[SUREFIRE-1760] Unbalanced number of test executions in forks (forkCount>1) results in no performance gain for jupiter/JUnit5 tests

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.22.1, 2.22.2, 3.0.0-M4
• Component/s: JUnit 5.x support
• Assignee: Tibor Digana
• Reporter: Vlad Skarzhevskyy
• Created: 07/Mar/20 03:32
• Updated: 08/Mar/20 03:55
• Resolved: 08/Mar/20 03:55

Description

Running JUnit5 tests using maven-surefire-plugin with  <reuseForks>true</reuseForks>  e.g. mvn -DforkCount=8

Results: number of tests in each fork is significantly unbalanced, some forks do not get tests at all..  

The end results is that on CI server the total test execution will take as long as longest fork and only 50% of the available resources are used. 

In our case we observed that our 3000 integration tests after migration to junit5 take 30 minutes insteads of 18 minutes as it was with junit 4.

The balance is restored when running with reuseForks=false  but this will cause additional overhead for complex integrations tests
     e.g. tests requiring additional setup , classpath scanning and so on..  

The results are similar with maven-surefire-plugin: 2.22.1, 2.22.2, 3.0.0-M4  
Maven 3.6.3 , openjdk version "1.8.0_242"  on Linux
 
BTW If junit-vintage-engine is used the same results observed for JUnit3 and JUnit4 tests.
 
Consider example of 100 tests  (10 classes 10 tests in each)
https://github.com/skarzhevskyy/range-junit/tree/master/range-junit5-surefire-fork
 
When running mvn -DforkCount=8  with extension that prints the number of tests executed in each fork getting following:

Tests completed  JVM Fork 8; total tests: 0; classes: 0
Tests completed  JVM Fork 4; total tests: 0; classes: 0
Tests completed  JVM Fork 1; total tests: 0; classes: 0
Tests completed  JVM Fork 6; total tests: 0; classes: 0
Tests completed  JVM Fork 3; total tests: 0; classes: 0
Tests completed  JVM Fork 2; total tests: 0; classes: 0
Tests completed  JVM Fork 7; total tests: 30; classes: 3
Tests completed  JVM Fork 5; total tests: 70; classes: 7

 
mvn -DforkCount=4

Tests completed  JVM Fork 2; total tests: 0; classes: 0
Tests completed  JVM Fork 3; total tests: 10; classes: 1
Tests completed  JVM Fork 4; total tests: 40; classes: 4
Tests completed  JVM Fork 1; total tests: 50; classes: 5

 
The distribution is also random,  results may vary between maven runs:
mvn -DforkCount=4

Tests completed  JVM Fork 4; total tests: 0; classes: 0
Tests completed  JVM Fork 2; total tests: 10; classes: 1
Tests completed  JVM Fork 1; total tests: 30; classes: 3
Tests completed  JVM Fork 3; total tests: 60; classes: 6

Issue Links

• duplicates: SUREFIRE-1725 Surefire in JUnit Vintage mode distributes tests very unevenly between forks, causing poor parallelism