[SUREFIRE-1055] Parallel JUnit does not run all test methods with parallel=classesAndMethods perCoreThreadCount=false useUnlimitedThreads=true and threadCountMethods specified

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.16
• Fix Version/s: 2.17
• Component/s: Junit 4.7+ (parallel) support, Maven Failsafe Plugin, Maven Surefire Plugin
• Assignee: Kristian Rosenvold
• Reporter: Chris Hansen
• Created: 21/Jan/14 15:50
• Updated: 13/Feb/14 13:57
• Resolved: 13/Feb/14 11:52

Description

Some test methods are skipped at random with the below configuration. When test methods are skipped in this way, only one method in the class runs. Running tests repeatedly with no code change often yields a different number of tests with each run (e.g. see the attached surefire-output.txt). Tests that take longer to run are less likely to be affected. This affects version 2.16 of Surefire and Failsafe equally.

Here is a simple test project which reproduces the issue: https://github.com/hansenc/SUREFIRE-1055
It has a few simple test classes with a naming convention for how many test methods are in each class (e.g. Methods4Test has 4 test methods).

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.16</version>
    <configuration>
        <parallel>classesAndMethods</parallel>
        <perCoreThreadCount>false</perCoreThreadCount>
        <useUnlimitedThreads>true</useUnlimitedThreads>
        <threadCountMethods>3</threadCountMethods>
    </configuration>
</plugin>