[SUREFIRE-2195] Surefire reports are not correct when running tests in parallel using JUnit5

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0-M7, 3.1.2
• Component/s: JUnit 5.x support
• Assignee: Unassigned
• Reporter: Debraj Manna
• Created: 14/Sep/23 10:18
• Updated: 20/Sep/24 19:13

Description

The issue has been discussed in the Maven mailing list. I am creating a bug here as suggested in the mailing list

Cross-posting it here for completeness

My junit-platform.properties looks like below

junit.jupiter.execution.parallel.enabled=true
junit.jupiter.execution.parallel.config.strategy=fixed
junit.jupiter.execution.parallel.config.fixed.parallelism=8
junit.jupiter.execution.parallel.mode.default=same_thread
junit.jupiter.execution.parallel.mode.classes.default=concurrent

I am observing the surefire reports that are getting generated are not correct

I have a test class named com.spotnana.servicetests.backoffice.invoicing.AirInvoiceServiceTest contains just a single test, testAirTicketing().

But in the report file, TEST-com.spotnana.servicetests.backoffice.invoicing.AirInvoiceServiceTest.xml.

I see like below

<testsuite xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report-3.0.xsd" version="3.0" name="com.spotnana.servicetests.backoffice.invoicing.AirInvoiceServiceTest" time="44.711" tests="2" errors="1" skipped="0" failures="0”>

…

  <testcase name="testCrossPnrRefund" classname="com.spotnana.servicetests.analytics.ingestion.pnr.AirServiceTest" time="5.789"/>
  <testcase name="testAirTicketing" classname="com.spotnana.servicetests.backoffice.invoicing.AirInvoiceServiceTest" time="33.331”>

It seems to be picking tests from some other class like AirServiceTest as in this case.

The console logs are also not correct. Please check the attached screenshot. In the attached screenshot of the console logs, it is saying Test Run 5 but the AirInvoiceServiceTest contains only 1 test. And also the report shows random data. For example, sometimes it says Test Run 5 , sometimes it says Test Run 2, etc.

I am observing it as part of a big project consistently with every run. I am trying to reproduce it in a small example. If I am able to do I will share it. In the mean time if some debug logs are needed, let me know I will be happy to share them.

Junit5 Version - 5.9.3

I am observing the issue with Junit5 Version 5.10.0 also. I tried reducing the junit.jupiter.execution.parallel.config.fixed.parallelism to 3 but there also I am getting the same problem.

Issue Links

• relates to: SUREFIRE-2252 Failsafe xml for a test class has results from another class