[SUREFIRE-1725] Surefire in JUnit Vintage mode distributes tests very unevenly between forks, causing poor parallelism

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-M3
• Fix Version/s: 3.0.0-M5
• Component/s: Junit 4.x support, JUnit 5.x support, Maven Surefire Plugin
• Assignee: Tibor Digana
• Reporter: Gareth Daniel Smith
• Created: 26/Nov/19 10:52
• Updated: 10/Sep/20 14:36
• Resolved: 05/Jan/20 02:49

Description

To reproduce this issue, please run the attached test case using mvn clean verify

You will see output similar to the following:

[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running My7Test
Test in process 32569
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.017 s - in My7Test
[INFO] Running My9Test
Test in process 32569
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.983 s - in My9Test
[INFO] Running My4Test
Test in process 32569
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.998 s - in My4Test
[INFO] Running My8Test
Test in process 32569
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.999 s - in My8Test
[INFO] Running My6Test
Test in process 32569
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.001 s - in My6Test
[INFO] Running My1Test
Test in process 32569
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1 s - in My1Test
[INFO] Running My5Test
Test in process 32569
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.999 s - in My5Test
[INFO] Running My3Test
Test in process 32569
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.997 s - in My3Test
[INFO] Running My2Test
Test in process 32569
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1 s - in My2Test
[INFO] Running My0Test
Test in process 32569
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.999 s - in My0Test

Notice how all 10 tests are being run on the same process. This is wrong because there should be 5 forks - and the tests should be evenly distributed between them.

If Junit4 is used instead of Junit5+vintage (see the commented dependency in pom.xml) then this issue does not occur (probably because Surefire uses a different code path in that case).

Please note that this issue was originally logged against Junit (https://github.com/junit-team/junit5/issues/2114) - but they suggested the problem lies within surefire.

Thanks for any help!

Issue Links

• is duplicated by: SUREFIRE-1760 Unbalanced number of test executions in forks (forkCount>1) results in no performance gain for jupiter/JUnit5 tests