[SUREFIRE-869] Parallel test execution on class level runs not all tests but some twice

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.12
• Fix Version/s: 2.12.1
• Component/s: Junit 4.7+ (parallel) support

Environment

Apache Maven 3.0.4 (r1232337; 2012-01-17 09:44:56+0100)
Maven home: /opt/local/share/java/maven3
Java version: 1.7.0_04, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/1.7.0.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.7.4", arch: "x86_64", family: "mac"
JUnit 4.10

• Assignee: Kristian Rosenvold
• Reporter: Martin Burger
• Created: 24/May/12 05:09
• Updated: 19/Jun/12 14:27
• Resolved: 19/Jun/12 14:27

Description

When I run JUnit tests in parallel on the class level, Surefire behaves somehow erratically. For instance, having test classes TestA and TestB, Surefire sometimes runs TestA twice, TestB twice at other times - but never TestA and TestB in parallel.

I attached a simple Maven project that helps reproduce this issue. Via mvn test -P sequential you can execute two test classes sequentially:

...
-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running mavendebug.TestAppFirstTest
-----------------> TestAppFirstTest.test()
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.036 sec
Running mavendebug.TestAppSecondTest
-----------------> TestAppSecondTest.test()
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec

Results :

Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
...

Running mvn test instead runs tests in parallel:

...
-------------------------------------------------------
 T E S T S
-------------------------------------------------------

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Concurrency config is parallel='classes', perCoreThreadCount=false, threadCount=4, useUnlimitedThreads=false
Concurrency config is parallel='classes', perCoreThreadCount=false, threadCount=4, useUnlimitedThreads=false
Running mavendebug.TestAppSecondTest
Running mavendebug.TestAppSecondTest
-----------------> TestAppSecondTest.test()
-----------------> TestAppSecondTest.test()
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec

Results :

Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
...

As you can see, TestAppSecondTest is executed twice, while test TestAppFirstTest is not executed at all.

Please note: I chose Blocker as priority as this caused (1) that our extensive test suite was not executed completely during the last weeks which we only discovered today, and (2) that caused tests that run against a database to fail with unexpected constraint violations just because those tests where executed twice in parallel.