Uploaded image for project: 'Maven Surefire'
  1. Maven Surefire
  2. SUREFIRE-1760

Unbalanced number of test executions in forks (forkCount>1) results in no performance gain for jupiter/JUnit5 tests

Agile BoardAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Duplicate
    • 2.22.1, 2.22.2, 3.0.0-M4
    • None
    • JUnit 5.x support
    • None

    Description

      Running JUnit5 tests using maven-surefire-plugin with  <reuseForks>true</reuseForks>  e.g. mvn -DforkCount=8

      Results: number of tests in each fork is significantly unbalanced, some forks do not get tests at all..  

      The end results is that on CI server the total test execution will take as long as longest fork and only 50% of the available resources are used. 

      In our case we observed that our 3000 integration tests after migration to junit5 take 30 minutes insteads of 18 minutes as it was with junit 4.

      The balance is restored when running with reuseForks=false  but this will cause additional overhead for complex integrations tests
           e.g. tests requiring additional setup , classpath scanning and so on..  

      The results are similar with maven-surefire-plugin: 2.22.1, 2.22.2, 3.0.0-M4  
      Maven 3.6.3 , openjdk version "1.8.0_242"  on Linux
       
      BTW If junit-vintage-engine is used the same results observed for JUnit3 and JUnit4 tests.
       
      Consider example of 100 tests  (10 classes 10 tests in each)
      https://github.com/skarzhevskyy/range-junit/tree/master/range-junit5-surefire-fork
       
      When running mvn -DforkCount=8  with extension that prints the number of tests executed in each fork getting following:

      Tests completed  JVM Fork 8; total tests: 0; classes: 0
      Tests completed  JVM Fork 4; total tests: 0; classes: 0
      Tests completed  JVM Fork 1; total tests: 0; classes: 0
      Tests completed  JVM Fork 6; total tests: 0; classes: 0
      Tests completed  JVM Fork 3; total tests: 0; classes: 0
      Tests completed  JVM Fork 2; total tests: 0; classes: 0
      Tests completed  JVM Fork 7; total tests: 30; classes: 3
      Tests completed  JVM Fork 5; total tests: 70; classes: 7

       
      mvn -DforkCount=4

      Tests completed  JVM Fork 2; total tests: 0; classes: 0
      Tests completed  JVM Fork 3; total tests: 10; classes: 1
      Tests completed  JVM Fork 4; total tests: 40; classes: 4
      Tests completed  JVM Fork 1; total tests: 50; classes: 5

       
      The distribution is also random,  results may vary between maven runs:
      mvn -DforkCount=4

      Tests completed  JVM Fork 4; total tests: 0; classes: 0
      Tests completed  JVM Fork 2; total tests: 10; classes: 1
      Tests completed  JVM Fork 1; total tests: 30; classes: 3
      Tests completed  JVM Fork 3; total tests: 60; classes: 6

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            tibordigana Tibor Digana
            vlads Vlad Skarzhevskyy
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment