[SUREFIRE-1953] Junit 5 in parallel execution mode confuses Surefire Console logs

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: JUnit 5.x support
• Assignee: Unassigned
• Reporter: Praveen Narala
• Created: 26/Oct/21 11:05
• Updated: 27/Oct/21 09:23

Description

When I use Junit 5 to run the tests in parallel (not using Surefire Forking), the console logs are confusing when the test failed.

As per Junit 5 documentation, when you use Junit 5 for parallelism, the tests will run using different Forking threads in the same process. And, when there is a test failure, Surefire is logging respective test failure logs. Instead, it is just picking random logs from different threads.

Would it be possible to get the right logs for failed tests?

 

Junit 5 Configuration from Surefire Plugin:

 

 

Tests run:
[INFO] --- maven-surefire-plugin:3.0.0-M5:test (default-test) @ api-test ---
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.test.abc.ABCTests
[INFO] Running com.test.abc.XYZTests
[INFO] Running com.test.abc.ATests
[INFO] Running com.test.abc.GetLogsTests
[INFO] Running com.test.abc.FoldersTests
[INFO] Running com.test.abc.NameTests

 

 

Example Logs from Console:

 

NFO |1026-103227268|ForkJoinPool-1-worker-3|abc.ABCTests||||||About to execute test [delegatedUserAccount] - GUID [random-guid] and data [[1] com.test.abc.ABCTestData@37589144]
INFO |1026-103227268|ForkJoinPool-1-worker-13|abc.XYZTests||||||About to execute test [getTestId] - GUID [random-guid] and data [[1] com.test.abc.data.XYZTestData@4c506080]
INFO |1026-103227268|ForkJoinPool-1-worker-5|abc.ATests||||||About to execute test [getFolderTest] - GUID [0e1141b0-3445-36f7-879a-2f12f3de87a7] and data [[1] com.test.abc.data.FoldersTestData@484d6599]
INFO |1026-103227268|ForkJoinPool-1-worker-7|abc.GetLogsTests||||||About to execute test [getIds] - GUID [random-guid] and data [[1] com.test.abc.data.GetLogsTestData@4cf26b0c]

 

When there is any test failure from the execution, we see logs from all different threads like below...

 

 

INFO |1026-103227268|ForkJoinPool-1-worker-3|abc.ABCTests||||||About to execute test [delegatedUserAccount] - GUID [random-guid] and data [[1] com.test.abc.ABCTestData@37589144]
INFO |1026-103227268|ForkJoinPool-1-worker-13|abc.XYZTests||||||About to execute test [getTestId] - GUID [random-guid] and data [[1] com.test.abc.data.XYZTestData@4c506080]
INFO |1026-103227268|ForkJoinPool-1-worker-5|abc.ATests||||||About to execute test [getFolderTest] - GUID [0e1141b0-3445-36f7-879a-2f12f3de87a7] and data [[1] com.test.abc.data.FoldersTestData@484d6599]
INFO |1026-103227268|ForkJoinPool-1-worker-7|abc.GetLogsTests||||||About to execute test [getIds] - GUID [random-guid] and data [[1] com.test.abc.data.GetLogsTestData@4cf26b0c]

 

Expected Behavior

It would be good to have console logs at the Test case level rather than picking up from random threads though we run tests using Junit 5 or Surefire Forks (JVMs).

 

Steps To Reproduce

Use Junit 5 Configuration from Maven Surefire Plugin v3.0.0-M5 with JDK11.

<plugin>
 <artifactId>maven-surefire-plugin</artifactId>
 <version>${surefire.plugin.version}</version>
 <configuration>
 <trimStackTrace>false</trimStackTrace>

 <includes>
 <include>**/*Test.java</include>
 <include>**/*Tests.java</include>
 </includes>
 <properties>

 <!--includeTags & excludeTags don't work with Surefire v3.0.0-M5.
 Same thing now works with groups (Groups are nothing but Tags)-> means all Included Tags,
 excludedGroups -> means Excluded Tags. While running the tests using maven command
 with the help of Tags,either we should use -DTags/-Dgroups-->
 <groups>${Tags}</groups>
 <excludedGroups>Skip, WIP</excludedGroups>
 <!-- Parallel execution using Junit 5 -->
 <configurationParameters>
 junit.jupiter.conditions.deactivate = *
 junit.jupiter.extensions.autodetection.enabled = true
 junit.jupiter.execution.parallel.enabled = true
 junit.jupiter.execution.parallel.mode.default = concurrent
 junit.jupiter.execution.parallel.config.strategy=fixed
 junit.jupiter.execution.parallel.config.fixed.parallelism=6
 junit.jupiter.testinstance.lifecycle.default = per_class
 </configurationParameters>
 </properties>
 <systemPropertyVariables>

 <!-- https://bugs.openjdk.java.net/browse/JDK-8212755 -->
 <java.locale.providers>COMPAT</java.locale.providers>
 </systemPropertyVariables>
 </configuration>
</plugin>

Run the tests using the maven command to run the tests in parallel using Tags...

mvn -Dgroups=TestingTag